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ABSTRACT 

Research investigated: 1) the quality of literature 
disseminated through an educational information channel, 2) its 
acceptance level, and 3) its effects upon decision-making. Some 102 
research documents were selected from the 1971 entries in Research in 
Education (RIE), the abstracting and microfiche dissemination service 
for unpublished literature compiled by the Educational Resources 
Information Center (ERIC) . Researchers assessed the documents' 
quality and educational decision-makers evaluated their practical 
utility. Results showed that the researchers rated the overall 
quality of the RIE documents as low, but that their accpetance by 
decision-makers was high. Decision- makers with less research 
sophistication tended to overrate the quality of the documents; the 
quality of the materials did not differ among the ERIC 
Clearinghouses, but was found to vary directly with the quality 
control level of the organizations which sponsored the research. The 
study, therefore, indicated that RIE often disseminated low quality 
information, which was likely to have harmful effects upon its 
intended audience. A system of rapid refereeing was suggested as a 
means of maintaining the positive advantages of RIE while reducing 
the negative consequences. (Author/PB) 
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ABSTRACT 



The q,uality of the literature disseminated through a major educational 
information channel, the acceptance levels of this literatwe, and its 
effects upon decision making and implementation in the educational systems 
are the m^^ov considerations of this research. 

One hundred and two research-oriented documents were randomly select- 
ed from the 1971 entries in Research in Education ( HIE ) , the • abstracting 
and. microfiche dissemination service of unpublished literature compiled 
by the Educational Resources Information Center (BRIC). These reports 
were presented to a group of Educational Research Specialists to determine 
their queJ-ity. The same reports were submitted to a group of Educational 
Decision Makers for an assessment of their quality and usefulness from 
the practitioner's point of view. By analyzing and comparing these as- 
sessments, an attempt was made to determine: (l) the quality of the 
information being disseminated' (2), the acceptance levels of . the reports 
among Educational Decision Makers; (3) the plans Educational Decision 
Makers ma,ke as a result of the information being disseminated; and (k) how 

these implementation plans are related to the quality of the information 

\ 

being dissemimted. r 

The results indicated that, while there were some high quality 
reports in RIB , the overauLl quality was rated low by the Research Special- 
is-is. On the other hand, the acceptance levels of these same reports among 
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Decision Makers was high. Thtis a significant disparity was found between 
the quality of the reports emd their acceptance .levels among educational 
practitioners. Specific information relating to actual decisions these 
practitioners would base on these reports was not adequately obtained in 
this study. However, it was inferred that the rapid dissemination of low 

qviality information was actually a disservice rather than an assistance to 

\ t ' ■ 
the Decision Makers. . ^ ij 

In addition, the results suggested that Decision Makers with lower 
degrees of research sophistication were more likely to overrate the quality 
of the research.. Althougji no differences in quality of research were found 
among the vajrious clearin^ouses , differences in quality were found to 
be related to the sponsorship of the paper. . Papers sponsored by organ- 
izations with higher quality control were of significantly higher quality. 

The results, therefore, indicate that RIE often disseminates low 
quality information which is likely to have harmful effects on its intend- 
ed audience. The major recommendation *for this study is to introduce 
a, system of refereeing into the RIE system. A rapid refer eeing system 
would retain the eidvantages of RIE 3 while reducing the negative side ef- 
fects found in the present study. 



INTRODUCTION 

Need for the Stxxdy 
Knowledge and information in education, as veil as in science and 
technology in general, have proliferated rapidly in recent years. Expen- 
ditures for research specifically designed to improve educational 
practices have increased greatly. In addition, large scale attempts 
are heing made to apply knowledge from other fields to educational 
theory and practice. ^ 

\. 

This proliferation of knowledge and information has been ac- 

/ 

compdnied in the educational community by considerable concern about 
the adequate dissemination and implementation of reports, findings, 
collections, summaries, etc., vAiich have been compiled. An increase 
in information accomplishes little unless it is disseminated to persons 
who need this information for use in their professional work. Further, 

the dissemination of information is useful only to the extent that the 

/ 

I 

information being disseminated is accurate, of high quality, and in a 
form which will be of optimal use to the recipient of the information. 

The present study examines the quality of the literature dis- 
seminated through a major educational information channel, the ac- 
ceptance levels of the literature among its recipients, and its effects 
upon decision making and implementation in educational systems. 

Overview of the Study 
Research in Education ( RIE ) represents the indexing and micro- 
fiche dissemination system of the Educational Resources Information 
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Center (ERIC). The primary purpose of is to dissendnate infor- 
mation which is not already available in some other published sources. 
For example, papers presented at meetings and conventions, technical 
reports, final reports of various funded research projects, and other 
reports vhich are not published by their authors might be disseminated 
through Rjffi. 

While rapid dissemination of educational information is desirable 
there is considerable evidence that much of what is disseminated is of 
poor quality. For example, Michael (l963)j Campbell and Stanley (1963), 
Wandt (1965, 1967), Scriven (1967), Asher (1969)? Mann (1969), Caro 
(1971), and Vockell and Asher (1972) have suggested that quality and 
reliability are often lacking, even in research published in the 
scientific journals. If the published research literaturq> which is 
subject to prepublication editing, is of questionable quality, it would 
seem that unpublished research literature, which is largely unedited, 
would he of even lower quality. 

The present study attempts to determines (l) the quality of the 
research information b^iing disseminated in Rffi, (2) the acceptajice 
levels of this information among eduq&tion&l decision makers, (3) the 
plans educational decision makers mak^ as a result of the information 
being disseminated, and .(^l*) how possible implementation plans are 
related to the quality of the research. 

The basic procedure employed in this study was to select a random 
/ group of RIB research documents and to submit these documents for 
evaluation to two groups of judges: (a) highly qualified specialists 
in educational research, who would determine the quality of the documents; 
and (b) educational practitioners who represent the audience expected 
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to make educational decisions based on the information disseminated in 
the documents (i.e., the audience to whom E$IC is directed.) 

Significance of the/ Study 

If the documents which are being disseminated by RIE are of high 
quality, then this dissemination service is providing a valuable service 
to educational decision makers. Rapid deliveiyof high quality research 
would enable educational practitioners to bring relevant information 
to bear on their problems much sooner than would be the case if they 
would have to wait for formal publication of the research. Thug 
practitioners could make use not only of information published by "big 
name" educators and authors, but also of well-evaluated practical 
implementations of research by other practitioners. 

If the research disseminated by RIE is found to be of low quality, 
a typical reaction to such a finding would be to argue that such dis- 
semination is a waste of time and money.- Educational practitioners 
C6un gain little from information which is trivial or inaccurate. It 
will be axgued in this study, however, that such dissemination of low 
quality information is not merely a waste of time and money,, but actu- 
ally .an adverse influence upon intelligent educational decision making. 

When a person* buys a new house, he does not have to be an archi- 
tectural expert: he feels he has a ri^t to rely upon the judgment of 
an architect to provide appropriate, soundj architectural engineering. 
The situation is similar for users of educational research. A person 
faced with making an educational decision has the right to assume that 
the research information with which he is provided as a basis for his 
decision is accurate. He should not have to be an expert on multi- 
variate analysis of variance to be able to make decisions about studies 



If 

which use this analytical procedure. If he. is faced with a decision 
and has inf ormaiiion available from a factor analytic study, his prohlem 
should not he to determine whether the factor analysis was properly 
done, hut rather to determine whether the results of the study are ap- 
plicable to his situation and what effect these results should have 
upon his decision. 

The dangers inherent in disseminating low quality research inform- 
ation, therefore, are accentuated to the extent that it can he shown 
that educational decision makers are unable to differentiate between 
good and bad research* The present study attempts to exairdne both the 
quality of the research disseminated through RIE and the likelihood 
that low quality research which is disseminated will be considered 
high quality information by educational decision makers. 
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REVIEW OF THE LITERATUEffi 

Jlsbujijilnatlon of Scientific Information 
A National Acfiuiemy of Science Cojnmittee (l970) has listed five 
levels to \*hich scientific information needs to be disseminated: 
(l) the scholar, (2) the practitioner, (3) the elementary or secondcury 
school teacher, {k) the policy-maker or administrator, and (5) the 
citizen. Similarly, Brady and Braiisconib (1972) list, in decreasing 
order of specialization and increasing order of "breadth and generality, 
four general commamitiea in need of technical information: (l) the 
scientific specialist (the researcher) , (2) the industrial engineer 
(the applier), (3) the planner, the policy maker, and the manager 
(the innovators and guiders), and (k) the public (the consumer, 
beneficiary, and victim) . All persons in these various categories 
have a right ajid need for such information, but they do not all need 
it at the same level of specification, detail, specialization, under- 
standing, or speed. In view of such varying levels of need and 
sophistication, it has become increasingly obvious that some outside 
organizational help, beyond the usual abstracts, yearly reviews, sum- 
marizing books, etc., is necessary for most users of scientific 
information to help them keep abreast of new developments in various 
fields of science and for their various purposes. 

Dissemination of research findings can be either formal or informal. 
Informal dissemination includes such procedures as discussions with 
colleagues, corresponding among interested parties, and conversations 

ERIC 
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and exchanges at meetings. Gorvey, Lin, and Nelson (1970) present 
some interesting coD$>arisons between such informal presentations in 
the physical sciences , the sociaJL sciences, and education and their 
transition into more formal dissemination through Journals. They 
find that patterns of information flow somewhat differently in these 
various disciplines. The communications structures in education and 
the social sciences are relal^ively incohesive as compared to those of 
the physical sciences. 

In formfiLL dissemination of scientific information through journals, 
a major role of review and quality- control has traditionally heen al- 
locfitted to on editor or board of editors or consultants. The National 
Academy *s Committee (l970) states: "The editor's task is to decline 
work which is duplicative, incompetent, incorrect, or totally pedes- 
trian. This set of editorial judgments is the backbone of the scien- 
tific information system. It protects the inexpert reader and those 
who provide research funds, while assuring scientists in the field 
that the published work has been performed with competence and that 
the findings are probably reliable." (p. 

Quality of Educational Infonnatlon 
Havelock (I969) presents a detailed study of the processes of 
innovation, dissemination, and knowledge utilization. He also examines 
the characteristics of the individuals and organizations which facilitate 
or inhibit these three processes. Although this and similar studies 
reviewed by Havelock provide essential information about the systems 
and linkages involved in the dlsseininatlon process, there Is little 
explicit consideration of the quality of information being disseminated; 
that it may be inaccurate or even have detrimental effects in the 
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dissemination processes as the result of such inaccuracies. 

There has long "been criticism of the adequacy of the research and 
evaluation literature - published in the education journals. Speaking 
in general of the era of pre*-USOE research support and dissemination, 
for exaairple, Michael (19^3 ), an editor of the Research Methodology issue 
of the Review of Educational Research , has suggested", perhaps somewhat 
harshly, that, "Probably, on the average, only 10 percent of published 
papers in educational Journals are worthy of being reported in the 
Review ." Similarly^ Scriven (1967) has maintained that in education 
". . .by minimum acceptable research standards, 95 percent of the work 
In the field... that is concerned with causal analysiSf is, either by 
theoretical or practical standards, invalM or trivial." Sieber (1968) 
reports a review of 250 comparisons of live and televised instructions 
contedned in 31 reports which showed that most of the comparisons were 
ill designed, used inadequate sainpleSj misinterpreted the data, or 
suffered from other serious flaws. 

WaJidt as Chairman of an AERA Committee on Evaluation of Research 
(1965, 1967) published a report on an emipirical study of the qtiality 
of research in education. He selected 125 articles which he determin- 
ed to be representative of the research articles published in the 
broad field of education journals in 1962. These also included as- 
sociated fields of child development, educational psychology, and 
sociology. He submitted these articles to 125 judges who were deemed 
experts in the field of educational research by an AERA panel. The 
judges were aisked to rate the articles "accept," "revise" or "reject" 
on the basis of their acceptability for publication in a jourxxal of 
educational research. Of these published articles, the judges rated 

ERLC 
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19^6 "accept," 1+1^ "revise/* and kof, "reject." The judges also gave 
rather detailed lists of specific shortcomings which occurred in the 
articles which they felt should he rejected or returned for revision. 

Cemphell euid Stanley (1963, p. 176) suggest, "Much research in 
education today conforms to a design in which a single group is studied 
only once, subsequent to some agent or treatment presmed to cause 
change .... such studies have such a total absence of control as to be 
of almost no scientific value." Glass (1968) states that unless 
evidence of the worth of a new procedure generated from a single case 
or group is supported by public and replicable evidence, then the sup- 
port for the ideas and procedures can only be appeals to authority. 
These authorities may be very able people, but it leaves the door open 
to self-interested persons, quacks, cuid frauds who to the public may 
seem to have equal, or even greater authority. Corey *s book. Action 
Research to Improve School Practices (1953) points out, "The results 
of many educational experiioents have been controversial because in- 
adequate provision was mado for obtaining data describing" their suc- 
cess or failure. Those who substitute for the *tried and true* newer 
and presumably more promising practices are undar an especial obliga- 
tion to obtain objective evidence about consequences (p. lOO)/' 

More recently, Campbell (1969? P- ^09) has indicated possible ^ 
political involvements accon^panying many educational innovations which 
could maJce accurate evaluations extremely difficult* "...most admin- 
istrators wisely prefer to Jimit the evaluations to those the outcomes 
of which they can control, particularly insofar as publish^^d outcomes 
or press releases are concerned. Ambiguity, lack of truly comparable 
comparison bases, and lack of concrete evidence all work to increase 
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the administrator's control over >?hat gets said, or at least to reduce 
the bite of criticism in case of actual failure. There is safety under 
the cloak of ignorance." Further, as Caro (1971, p. 91) indicates, 
, "Those who actually carry out the programs to be evaluated are subordinate 
to those to whom the evaluators report . The issues addressed by the 
evaluation and the manner in which results axe reported aire strongly 
related to sponsorship. Consequently, the interests of the general 
public, practitioners, and recipients of services are not often fully 
served by evaluators." 

Another difficulty arises out of the demand for quick results* 
Mann (1969, p, 13) states, "The better the study, the longer it takes, 
and consequently the less usefulness it may have. Conversely, the 
sloppier the procedure, the more likely it is to provide information 
on questions of interest even though this data will be of doubtful 
validity." 

Quality Control in Research Dissemination 
The iinplications of infoxmtion quality for educational evaluation 
are proposed by Stufflebeam, Foley, Gephart, Guba, Hammond, Merriman, 
and Rfovus (1971, P- 61-63) in their evaluatlonal model. It distin- 
guishes among four general types of decision settings: (l) decisions 
to effect large changes supported by a high level of relevant inform- 
ation grasp, (2) decisions involving small changes and high inform- 
ation grasp, (3) decisions involving small changes and low information 
grasp, and (k) decisions involving large changes and low information 
grasp, (a change is "large" if society considers the variables to be 
altered important and if society considers the proposed change to be 
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important rather than trivial.) The depth and quality of information 
demanded from the dissemination process will vary f^rom setting to set- 
ting. Decisions which involve large changes will demand more detailed, 
accurate information than those involving smaller changes. Stufflebeam, 
et al. state that for educational practitioners "evaluation must he 
more extensive when there is only little information (or when the 
client cannot use availahle information in its present form)." 

Asher (1972) . states that it is not possible for professional 
educators to read the literature in education and the related journals 
without a good knowledge of the theory of measurement, internal validity 
concepts of research design, and some statistics. This problem is more 
serious in education than in the physical end biological sciences. One 
reason may be that engineers and physicians are trained in the measure- 
ment principles and research methods of the physical or biological 
sciences, while most professionals in education ore not so well trained 
in the behavioral sciences. In addition, comments, critiques, euid 
rejoinders ore a standard part of the literat\:re in the physical 
sciences, whereas this is not the case in the behavioral sciences. 
Vockell and Asher (l972) determined the 15 most frequently cited jour- 
nals in the 19^9 Encyclopedia of Educational Research and sent question- 
naires to the editors of these journals to determine policy with regard 
to critical comments. The results clearly indicate that theae Journals 
do not as a general practice contain critiques, comments, rejoinders, 
or critical letters to the editor . 

Kronick (19^9) states that the scholarly Journal originally 
developed as an open record in which each scholar submitted his find- 
ings to his fellow scientists for their review and criticism. Such 
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exchange of critical comment is looked upon as an essential part of 
the scientific dissemination process. Garvey emd Griffith (1971, p. 357) 
have reported on a study of psychologists ^Aio had distributed pre- 
prints of their scientific papers: "Over 605^ of these authors received 
feedback that prompted them to modify their manuscripts. These modi- 
fications were not simply a matter of improvement ^in graaimar and style 
of the manuscript but, instead, involved significant modifications 
such as reanalysis of data, redefinition of concepts, etc." The 
importance of good refereeing and q,iiality control at the editorial 
level is magnified by the fact that teacher-oriented organizations 
seem to be initiating publication policies ^ich will put "practical" 
information rather than reseetrch studies into t'Ue hajids of classroom 
teachers. For example, Riedesel (l97l) indicates that the future 
policy of The Arithmetic Teacher, a journal of the Rational Council 
of Teachers of Mathematics (NCTM) , willijbe not to publish research 
articles. Boplicit in such a policy change is the expectation that 
readers can implement what they find useful without bothering^ with 
the details of the research design. 

Gideonse (1968) presented an output-oriented model of reseeurch 
and development elaborating and extending Guba and Clark *s (1965) 
well known model. He attea35>ts to classify the processes related to 
cuid necessary for change in education. The first stage is Development, 
which includes both invention and design of the iimovation; the second 
stage, Diffusion, includes dissemination and demonstration. (Gideonse*s 
definition of Demonstration includes such processes as "to examine and 
assess" the qualities of the innovation.) The final stage is Adoption, 
which incloides trial, installation, and institutionalization. Here 
O part of the objective is, '"To. , .provide a basis for assessing the 
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quality, value, fit, and utility of the invention. . .i .e . , to test." 
Thus Gideonse considers systematic evalue-tion of the material to "be 
disseminated as an integral part of the Development and Dissemination 
process. 

Asher (1972) . states, "...lack of systematic resecurch editing, 
critiques, and comments in the literature .. .hurts in a non-negative 
way as well. It is often stated that everybody knows the quality of 
the literature in education, and this is particulaxly true of doctoral 
dissertations. But there are well done studies in the literature of 
value to education that are ignored. .. .Again the total volimoe of the 
literature is so laxge and generally of such poor quality in terms of 
its objectivity that often the strategy in using it is to implement 
those research ideas which occurred most recently, that is, what is 
most popular at the moment . This practice haxdly "builds enduring 
qvialities in an educational system. Meaaiwhile communications and infor- 
mation of high quality tend to "become o"bliterated in the mass of docu- 
ments." Asher recommends as a partial solution an explicitly defined 
Elimination process added to the Gideonse (1968) model. 

Garvey ar/l Griffith (l97l) view this pro"blem from a slightly 
different viewpoint. They maintain that an Informal information system 
must exist alongside the formal dimension, but they point out that this 
informal domain is not meant to have the same attributes as information 
published in the reputable journal literature. These authors maintain 
that scientists and authors need a forum where they can present half - . 
formed ideas for critical commentary from their peers before re- 
evalmting them and presenting them to the public. Such information 
and ideas may turn out to be completely false, but as long as they are 

O 
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contained in the informal domain, they shoiild te relatively harmless. 
The potential difficiilty is that such informal, information may be dis- 
seminated in such a way that . .regardless of efforts to keep these 
reports from taking on the status of a formal communication, the 
receiver gets the impression that the reported findings are sound, and 
the discovery is original. This conclusion may be reached singly 
because the reports no longer have the characteristics of informal 
communication and come to reseirible the formal journal article (p. 361)." 

Contputerized Information DisseaaodLnation Systems 
The coinputer has introduced new advantages and opportunities for 
the storage and dissemination of scientific information. Such advantages, 
however, are accompanied by difficulties and responsibilities. The 
work of Garvey and Griffith (I963, I96lfa, 1961fb, I965, 1966, 1967> 
.1971) has shown that concentration on retrieval techniques alone is 
not ail adequate way to improve a communication system if the desired 
outcome is a net increase in the amount of relevant knowledge avail- 
able to the working scientist (Clark, 1971) • An excessive volume of 
information tends to swamp the readei!.^{Licklider, 1966) . Holt (1971, 
P» 331) points out that "The undoubted merits of computers in this 
kind of work seem to have made many of us overlook the fact that the 
pr6ble33[is we face are not primarily technological. .. .It iis becoming 
evident, however, that the diagnosis of our communication problem is 
mainly ^ixiformation input overload} and the strategy of coping with 
it that seems called for is reducing the amount of input by better 
control of quality." 



ERIC 



With the incrrease in voliojie of information, there is greater dif- 
ficulty for the inexperienced reader in determining ^at is good and 
what is had in literature. Ziman (1969, p. 319) points out that "We 
must he ahle to rely on the basic accuracy and honesty of what we read 
in other people *s papers, for we are always using their results in the 
construction of our own researches, and simply cajinot find the time to 
repeat all their experiments, measvirements, calculations, for our- 
selves....! cannot see how this innocence could he preserved against 
careerist pressures to publish, if there were no scrutiny by e:jqpert 
referees." The National Academy of Sciences Committee (1970) adds 
that even when a fully computerized information retrieval system is 
eventually developed "...the role of the editors and reviewers will 
remain unchanged: indiscriminate release of unedited reports to a 
conputer network could well be even more disastrous than indiscriminate 
publishing would be today." Loevinger (1972, p. 9) adds that . 
"Propogation of errors is far more detrimental to science than a 
moderate delay in the propogation of truth." 

Educational Resources Information Center (ERIC) 
An example of the application of computerized retrieval and dis- 
semination systems to the field of education is the Educational Resources 
Information Center (ERIC) . ERIC can be described as a national infor- 
mation system designed and supported by. the National Institute of 
Education. Its purpose is to provide ready access to results of 
"exemplary programs, research and development efforts, and related 
information that can be used in developing more effective educational 
programs" (ERIC, 1970b), ERIC employs a system of specialized clear- 
ing houses, each of which is responsible for a particular educational 



15 



area. Current significant informtion relevant to education is 
monitored > acquired, evaluated ^ abstracted > indexed, and listed in 
ERIC reference sources. Throu^ these reference sources, **any educator, 
anywhere in the country, has easy access to reports of innovative 
programs, conference proceedings, bihliographies, outstanding profes- 
sional papers, curriculum-related materials, and reports of the most 
significant efforts in educational research and development, regardless 
of where they were first reported" (ERIC, 1970b). 

ERIC has two major indexing and abstracting publicationa: 
Curr_ent Index to Journals in Education ( CUE ), which indexes published 
information in education; and Research in Education ( RIE ), which 
indexes information which has not been formally published in a scien- 
tific or professional journal. A r':5cent (l970a) ERIC newsletter 
states: - .ERIC*s two principal publications, RIE auid CUE axe 
remarkably efficient guides to what is good in literature. Putting 
ourselves in the shoes of potential users, we sagaciously clear only 
the best for inclusion in RIB Computerissed indexing of what is 
available in ERIC will facilitate your 'ferreting out* as no other 
information system has ever done • " 

It would seem that the RIB aspects of ERIC may present a contra- 
diction. If the published literature is of the quality Wandt, Camp- 
bell, Caaiipbell and Stanley,. Glass, and Corey suggest that it is, what 
value can there be in the massive dissemination of unpublished liter- 
ature, which has only minimal review by scientists and professionals? 
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Quality of the RIE Resesirch 

An investigation was initiated at Purdue University (Asher & Vockell, 
1970, unpublished) to attempt to evaluate the quality of the informa- 
tion available througji RIE . Twelve judges were selected who had taken 
advanced courses in research design, statistics 3 and measurement tech- 
niques. A total of 7ii5 citations were randomly selected from RIE and 
were assigned to the judges. The judges were instructed to (a) determine 
the number of reports cited involving any evidence of empirical data to 
support the information included in the report, and (b) evaluate the 
quality of the articles designated as "research aa:'ticles" according to 
predetermined criteria. 

Of the 7^5 reports investigated, a total of only 155 were classi- 
fied as data-oriented resecurch. Of these> 22 w^re not available in 
either microfiche or hardback from ERIC's reproduction service and there- 
fore covild not be investigated further. Thus a total of 133 (l7-3^) of 
the original 7^5 citations referred to available research reports. 

The judges then were asked to undertake three forms of . evaluation: 

(a) rank the reports on a 5-point scale according to the quality, 

(b) rajik the reports "accept," "reject" or "revise" according to whether 
or not the articles should be accepted for journal publication, and 

(c) list the possible errors which appeared in reports listed either 
as "reject" or "revise." A summary of the results is given in Table I. 

As the results indicate, a total of only 6.7^t of the original 7^+5 
citations (39^ of the research documents) referred to reports which 
were supported by research which is either good or excellent. A total 
of only 2.6^ of the original 7^5 citations (15^ of the research reports) 
referred to reports which were considered worthy of publication as they 

ERIC 
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were written • These low ratings would seem to te the result of the pre- 
viously cited looseness in the infonnal communication system and in eval- 
uations sponsored "by the same organization i^rtiich is being evaluated • As 
suggested hy Garvey and Griffith (1971), a serious problem can arise when 
such information is transformed into a formal communication system, 

TABLE 1 

QUALZri RATINGS OP RE DOCIMIIITS (Afiher & Vockell, l^O) ' 



■ 7 


No. 


Cum ^ 

Research 

Reports 


Cum i 
All 

Reports 


Excellent— A laodel of good practice 


19 


9.8 


1.69 


Good— A few Mnor defects 


39 


39.3 


6.76 


Mediocre— Not good, not bad 


33 


64.3 


11.05 


Poor— Some serious defects 


37 


92. k 


15.86 


Cotnpletely incompetent— "horrible" 


10 


100.0 


17.16 




132* 






Accept , 


Revise J 


Reject 




Accept 


20 


15.0 


2.60 


Revise 


k8 


51.1 


. 8.84 



Reject 65 100-0 " 17-29 

i 

* » 1 report not rated 

Dissemination is an integral and important aspect of scientific and 
professional enterprises in general euid education and educational research 
specifically- ERIC is a major effort in educational research 'dissemination 
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process. Yet, the dissemination process is tased in great part on the 
assumption that what is heing disseminated is factual and worth dissem- 
inating. In the light of the research reviewed eind conducted it seems 
that this might he a somewhat tenuous assumption. 

Research Evaluation Instruments 
Research has been reported on the development of various objective 
instruments. for Judging the quality of educational research and evaluation. 
Bartos (1969) reviews 32 such instruments. Weuidt (1967) in his previously- 
cited AERA work developed a list of 25 characteristics with a five-point 
quality-rating scale for each characteristic. Wandt also included an 
"accept-revise-reject" scale and a system for classifying individual 
errors pointed out hy the Judges. Persell (1966) used a similar combina- 
tion of individual ratings of specific characteristics combined with an 
1 

overall rating of the quality of the work, based on both the substance 
QXid tl^e methodology of the research. Suydam (1968) has developed an 
instrvunent for evaluating experimental educational research reports, 
based on an analysis of nine general areas which results in a cumtulative 
overall score for the report. A similar instrument was developed by Kohr 
euid Suydam (l970) for evaluating survey research. Checklists and principle?? 
which caa serve as guidelines for evaluating educational research and 
evaluation can be found in Farqixahar and Krumboltz (1959)? Borg (1963)3 
Campbell and. Stanley (1963), Mpuly (1963)5 Ruramel (l96i^), Travers (l96if), 
Kerlinger (196H) > Van Dalen (I966) , Scriren . (I967) , . Suchman (1967), 
Grobman (1968), Fox (1969)3 Gephart and^Bartos (1969)3 Baxter (1970), =aB<3. 
Stufflebeam, et al. (l97l). McReynolds (l97l) presents research on the 
reliability of such ratings of research papers. 
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Statement of the Problem 
The central problem of this study is to determine the quality of t^e 
literature disseminated through a major educational dissemination channel, 
its acceptatnce levels among Educational Decision Makers, and the possible 
effects of its implementation in educational systems. Specifically, the 
following questions will he invest i^ted; 

1. What is the quality of the research information in this major 
educational dissemination channel? 

2. How does this quality compare with that of refer eed and edit- 
ed research literature which is formally published in prof- 
essional Journals? 

3. Does the quality of the literature vary among different ERIC 
clearinghouses? 

k. Are RIB documents which have been submitted to some degree 
of refereeing of better quality than those which have been 
submitted to little or no refereeing? 

5. What axe the acceptance levels of these disseminated reports 
among the audience of educational practitioners, administrators 
teachers, and developers? 

6. Are there differences in research sophistication among Educ- 
ational Decision Makers vftxich are related to their evaluations 
of the research documents? 

?• What implementation plans would these educational practition- 
ers make as the result of the information disseminated in 
this channel? 

8. How are these implementation plans related to the previously 
determined quality of the information disseminated? 
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Are there differences between ratings by Educational Research 
Specialists and ratings by Educational Decision Makers which 
are related to the nature of the decision situation? 
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PROCEDURES 
Qverviev 

In the present study, 102 research-oriented documents were randomly- 
selected from ERIC*s Research in Education . These documents were each 
assigned to an Educational Decision Maker and to an Educational Research 
Specialist to be reeid and evaluated. The Research Specialists were ex- 
pected to rate the dociiments according to the quality of the research in- 
formation, and the Decision Makers according to its usefulness in their 
decision making processes. 

The results of the evaluations of the two groups were tabulated and 
compared to determine (a) the quality of the research contained in the 
documents, (b) whether discrepancies existed between the ratings of the 
two groups, (c) the kinds of decisions Educational Decision Makers would 
make based on these reports, and (d) the relationship between the prob- 
able decisions and the quality of the research. 

In addition, the dociMents were subdivided into groups according to 
the amount o^ referee ing they had received prior to dissemination, and 
these groups of documents were compared to determine if differenceti in 
qizality existed. The documents were also subdivided according to the 
level of sophistication of the Decision Makers who reviewed them, and the 
ratings of these documents were compared to determine ^Aiether the dis- 
crepancies between ratings of Decision Makers and rating of Research 
Specialists were related to the Decision Maker's level of sophistication. 

ERLC 
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Each of the components mentioned atoovp will he descrihed in detail 
in suhsequent sections • 

The Documents in the Study 
A five percent random saiDple vas drawn from all the docuiiients indexed 
and ahstracted in ERIC*s Research in Education ( RIE ) diaring 1971. Five 
two digit nunibers were selected from a tahle of random numbers, and all 
documents whose accession numbers ended in these two digits were included 
in this init?.al sanple. This yielded a total of 617 citations. Next the 
abstracts of these documents were examined, and each report was classi- 
fied as either research-oriented or non-research-oriented. A report was 
classified as research-oriented if it contained empirical data which were 
used either to evaluate a program or to test an hypothesis. Otherwise it 
was classified as non-research-oriented. Thus experiments, quasi-experi- 
ments, scientific surveys, field stiidies, and scientific case studies were 
classified as research-oriented. Bihliographies, reviews of the litera- 
ture, census or status surveys, policy statements, personal opinions 3 cur- 
riculum guides, administrative guidelines, and other documents which con- 
tained no supporting empirical data were classified as non«research-orient- 
ed. . 

This classifying was done hy two independent judges. Initial clas- 
sifying provided agreement on all but thirty documents. The abstracts of 
these thirty documents were re-examined, and the full documents were 
obtained when available on microfiche. On the basis of the examination 
of this further information the same two judges again classified the 
documents. This second attejmpt at classification resulted in agreement 
on all but five documents. These five documents were examined by a third 
judge and were classified as resesirch-oriented or non-research-oriented 



23 

on the "basis of the agreement of two out of the three raters. This process 
of classification resulted in 203 of the original 6l7 citations being 
classified as research-oriented. 

Forty-one of these 203 research-oriented documents were listed in 
Research in Education as not available from the ERIC Dociament Reproduc- 
tion Service (eDRS) in either microfiche or hardcover format. Another 21 
were available in microfiche but not in hardcover format. Thus of the 
origirial 6l7 documents in the initial san5)le of dociunents, only ikl were 
research documents in the format needed for mailing to the reviewers. 
From these, 102 documents were randomly selected and were purchased from 
EDRS for subseq.uent analysis in the current study. (The documents were 
numbered from 1 through ikl. Thirty-nine three-digit numbers within this 
range were selected f!rom a random numbers table, and the corresponding 
39 documents were omitted from the study.) The titles of these 102 
documents and their RIE accession numbers are given in Appendix A. 

The Educational Decision Makers 
The Educational Decision Makers consisted of professional educators 
in elementary and secondary school systems, in state educational agencies, 
in colleges auid universities, and in other education-related fields which 
are part of ERIC*s audience. A person was classified as a "Decision 
Maker" if he was an educator at a level higher than a teacher of a single 
classroom and conceivably could make aua educational decision based on the 
information in the report he would review. All Decision Makers; were 
persons who had volunteered to take part in the study and wiio were told 
that they would receive an honorarium ranging from $15.00 to $30.00 for 
their services. Initial contact was made with these Decision Makers 
either by letter (see Appendix B) or by phone. (The phone message conveyed 

,9^"She same ixiforraation contained in the letter.) 
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VJith a few exceptions, the Decision Makers were selected from Indiana 
and sitrrounding states and metropolitan areas. This procedure was fol- 
lowed for several reasons. First, such a sample was geographically con- 
venient for researchers from Purdue University, ^Aio were atle to make the 
necessary personal follow-vcps required "by the study to obtain an except*> 
ionally hi^ rate of return and to obtain necessary clarifications and 
interpretations of results. Second, a population of Educational Decision 
Makers was readily available from up-to-date lists from state, city, and 
other educational agencies* Third? it appears that Indiana and geograph- 
ically close major metropolitan areas should not be considered atypical of 
school personnel, systems, and institutions throughout the Iftiited States. 
Indiana is known as the "crossroads of the nation*,' and there is evidence 
from Project TALENT that Indiana high school students are quite typical 
of U.S. students in aptitude and achievement (Asher & Dodson, 1972, in 
press) . 

A list of Educational Decision Makers was obtained of participants 
in the Model Training Project (MTP) in Educational Evaluation, a nation- 
wide program centered at Ohio State Uni veracity. The MTP trains and re- 
trains personnel in evaluation techniques to be eniployed as part of the 
decision making and change processes. Additional names of Decision Makers 
were obtained from the Indiana State Departme>at of Public Instruction and 
from Phi Delta Kappa. These prospective participants were contacted by 
letter (Appendix B) to solicit their participation.. Participants listed 
areas of professional interest , - and research reports were then matched to 
these areas of interest. 

The documents not placed through this process were then listed and 
attempts were xoade either by letter of phone to obtain reviewers who would 

ERIC 
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be interested in the specific area covered by each document. Lists trctm 
various departments of education, faculty lists from universities, and 
recommendations from personnel in these two groups were consulted* The 
potential reviewers were contacted auid their participation was solicited. 
This procedure resulted in the event'jal placement of all the dociiments. 

The names eaid professional positions of all the Educational Decision 
Makers are listed in Appendix C 

The Research Specialists 

The list of 166 reviewers employed by Wandt (1967, p. 32) was obtain- 
ed, and an attempt was made to mail a letter to all of these reviewers 
to solicit their participation as reviewers of ERIC reseeurch-oriented 
documents. A copy of the letter is included in Appendix D. These review- 
ers had been selected by Weuidt on the basis of professional expertise. 
At the time of his study all were AERA members, 98io had doctorates, 63% 
had taught research methods courses, 75^ had supervised Ph.D. work, sjnd 
their median number of professional publications was 16. 

Thirty-five of the original I66 reviewers contacted replied that 
they would be unable to participate. Reasons for non-participation were 
not activeiy requested, but when these were volunteered they ranged from 
overly complicated work schedules to unwillin^iess to ^'participate in 
OE enterprises." One hundred reviewers replied that they would be will- 
ing to read and evaluate a docvunent* Two reviewers were xmable to parti- 
cipate, but recommended specific colleagues they felt were qualified to 
take their places, auad these suggestions were accepted. Three reviewdrs 
were known to have died since the Mandt stxidy. The other 26 reviewers 
could not be contacted or failed to respond after five months, and no 
further information was obtained on them. 

ERIC 
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Attempts were made to send reviewers docunients close to their field 
of interest. In the course of the study it teceune apparent that several 
of the documents were completely outside the areas of expertise of the 
reviewers available. For exajirple, noreof the reviewers felt qualified 
to evaluate a document on nursing education. In such cases, persons from 
the content axea of the report who were also highly qualified in research 
skills were asked to participate. Their qualifications as researchers 
were based on their published research in the designated content area and 
the recommendation of a Research Specialist already in the sanrple. Three 
additional reviewers were obtained in this memner. The names of all the 
Research Specialists are included in Appendix E. 

To further ascertain the qualifications of these reviewers as Research 
Specialists, the reviewers were asked at the time they were con5)leting the 
evaluation form (a) whether they were ci;a?rently teaching a course in 
research methodology or had ever taught such a course, and (b) how many 
research reports they had published in professional journals or presented 
at professional meetings. The precise wording of the question used to 
obtain this information is included as part of the questionnaire in Ap- 
pendix J. ' 

Since the primary fimction of these reviewers was to read and 
evaluate the reports assigned to them fl*om the point of view of research 
quality, these reviewers will be referred to as "Research Specialists" 
in subsequent portions of this report. 

Evaluation Instrxaments 
The Educational Decision Makers completed the Information Quality and , 
Uses Pomi (Appendix I) . The techniques employed in analyzing the responsec 
to this evaluation form are explained elsewhere in this report. The 
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rating scales, which are the priinary sources of data in the present study, 
are labeled A, B, C, D, and E. These labels were not on the copy of the 
evaluation form sent to the reviewers, but are added here to facilitate 
readability. The Decision MaJkers also conipleted the General Info3rmtion 
Pom (Appendix L) . 

The Research Specialists coD?)leted the Form for Evaluating f^lity 
of Research (Appendix J) . This form differs from the Information Quality . 
and Uses Form in two ways: (l) the^ references to decision making are 
omitted, and (2) additional leading questions were added as part of item 
three. The rating scale items correspond to the similarly labeled items 
on the Decision Makers' evaJLuation form. These overlapping items are the 
main basis for comparison of the two sets of evaluations. 

Both forms contatined open-ended questions regarding the educational 
significance and possible methodological shortcomings in the document 
being evaluated. The Information QueuLity and Uses Foi*m also contained 
two seven-point rating scale items concerning the nature of the decision 
setting and an open-ended question on possible plans for implementation 
of the research results. 

Information on the reliability and intercorrelations of the items on 
the rating scales will be found elsewhere in this report. Prior to their 
use in the present study, all forms were administered, with satisfactory 
results, to an advanced educational research class to determine the clarity 
and readability of the formats. 

Evaluation of Documents by Decision Makers 
A copy of one of the documents in the sample, matched to area of 
professional interest, was mailed to each of the Bducational Decision 
Makers who had volunteered to participate . The Decision Makers 
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were instructed to read the document and to return the completed evalua- 
tion form within a week to ten days . A copy of the cover letter which 
accompanied the documents is contained in Appendix F. 

If the evaluation of a document was not returned within three weeks 
after the document had been sent to the Decision Maker, he was contacted 
either "by telegram or by phone. At this time he was asked if he could 
complete the evaluation in the immediate future. If this was impossible, 
he was asked to return the document so that it could be sent to a dif- 
ferent reviewer. A sample telegram is contained in Appendix G. If a 
document was returned imreviewed by the Decision Maker, then the partic- 
ipation of a new Decision MRJ^^er interested in the content of the document 
was solicited according to the same procedures employed in the selection 
of the original reviewer, and the document was immediately sent to this 
new reviewer. 

Evaluation forms were returned by mail in stamped, self -addressed 
envelopes which accompanied the dociunent when it was sent to the reviewer. 

Evaluation of the Documents by Research Specialists 
Each of the Research Specialists received a document to read and 
evaluate. Since the lists of both the doctmients and the Research Special- 
ist were predetermined, it was much more difficult to match areas of 
interest in the case of the Research Specialists than in the case of the 
Decision Makers. The Research Specialists were instructed to return the 
document they had received if they found it was entirely outside their 
area of professional interest and expertise. When documents were return- 
ed for this reason, the document was reassigned to different Research 
Specialists. Since the number of available Research Specialists was 
limited, this procedure made it necessary for several readers to read and 
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evaluate a second docuiiidnt. An attempt was mde to reassign a different 
document to reviewers ^o had returned their original document, "but if 
this was iinpossible it became necessary to drop the Research Specialist 
from the sample of reviewers, (in such cases, the names of the Research 
Specialists tAio were thus eliminated are exclixded from Appendix E.) 

In addition to the evaluation form contained in Appendix J, the 
Research Specialists received three supplementary evaluation checklists 
(Appendix K) which they were encouraged to consult to whatever extent 
they felt advisable. These supplementary checklists (which were color- 
coded to reduce confusion with the evaluation form to be completed as 
part of the study) were for the benefit of the reviewers and were not to 
be returned with the evaluation form. These supplementary checklists were 
sent only to the Research Specialists, not to the Decision Makers- It 
was felt that sending these to the Decision MeJcers would artificially 
focus their attention on methodological concerns, whereas what was desired 
was a "typical" reaction of the Decision Mak€.T, Specific focusing on 
methodology was felt to be beneficial in the case of the Reseaxch Special- 
ist, since this was their area of expertise. 

A copy of the cover letter which accom;panied the document is contain- 
ed in Appendix H. 

If a document was not returned within three weeks, the reviewer was 
contacted by telegram or telephone. This procedure was identical to that 
for the Decision Makers, with the exception that if a document was return- 
ed, an attempt was made to reassign the document to a Research Specialist 
already within the saniple^ rather than soliciting the participation of an 
additional reviewer. 

Evaluation forms were returned by mail in stamped, self -addressed 
envelopes which accompanied the document when it was sent to the reviewer. 
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Reliability of the Rating Scales 
Ten of the dociaments were initially selected for the reliability 
sample . Each of these documents was sent to a second Decision Maker and 
to a second Research Specialist. The instructions for these reviewers 
were the same as for the first set of readers. Correlation coefficients 
were computed "between the ratings of the first group who read each document 
ajid the group who read the same documents as part of the reliability 
ssjople* 

Two additional documents were inadvertantly sent to a second 
Research Specialist, and one additional document was sent to a second 
Decision Maker. (This happened because reviewers who had originally said 
they would be unable to return their completed evaluation forms by the 
designated deadline, eventually did return their forms.) These documents 
were included as part of the reliability saarple. Thus, the reliability 
sample actually consisted of twelve documents evaluated by two Research 
Specialists and eleven documents evaluated by two Decision Makers . 

The documents used in the reliability study are accompanied by 
asterisks in Appendix A. Similarly, the names of the Decision Makers 
and Research Specialists who took part in the reliability study are 
accompajiied by asterisks in Appendices C and E. 

The second documents in the reliability sample were used only in the 
coniputatipn of reliability coefficients. It was felt that double use of 
these documents would result in giving inappropriate additional weight to 
them. Thus, each of the 102 documents was used only once in all non- 
reliability analyses. 
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Axxalysis of the Data 

The primary data to be analyzed were the ratings by the Decision 
Makers and by the Research Specialists on the Rating Scales at the ends 
of their evaliiation forms. These data were analyzed both by frequency 
counts of the nmber of responses in each category on each scale and by 
analyses of these ratings after they had been converted to an appropriate 
1-2-3-^-5 or 1-2-3 interval scale. 

It is important to note that the direction of rating is reversed on 
Scale A of both the evaluation fonns. A rating of "5" on this scale is 
an indication that the reviewer viewed the document to be of high quality. 
On all other scales, a higher rating indicates lower quality in the 
document being reviewed. Thus "l" was applied to "Accept^, " and "3" was 
applied to "Reject." To prevent confusion, footnotes indicating this 
irregularity acconrpany all figures and tables where' Scale A is involved 
in the analysis. 

The data were coded' and trajisf erred to conputer cards. Analyses were 
performed on the CDC 6500 computer at the Purdue University Computing ' 
Center. Frequency counts were performed using the BMD OkD and FREQ 
programs (Dixon, 1968) . The analysis of variance program was ANOVAR 
(Veldman, 1967)- ANOVAR is suitable for both one-way and repeated measures 
analysis of variance, and yields as output means for groups and trials, 
Meaji Sqxaares, F-ratios, degrees of freedom, and probability levels. Means, 
standard deviations, and correlation coefficients were computed through 
the BMD 03D program (Dixon, 1968). The tables in Guilford {l9C>k) were 
consulted to determine significance levels of correlation coefficients 
and t. tests. 
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To answer the first research question (What is the quality of the RIB 
research literature?), frequency counts and means were coinputed for each 
of the rating' scales. This was performed sepeirately for the Research 
Specialists and for the Decision Makers, providing ratings of the quality 
of the research information as viewed "by "both groups of reviewers. 

To answer the second reseeirch question (How does this quality compare 
with that of published literature?), the meaai ratings on the Accept-Revise- 
Reject Scale were compared to the parallel data compiled hy Wandt (19^7 ) 
throu^ a t test. 

To answer the third research question, the reports were grov^ped ac- 
cording to the clearinghouses from vAiich they originated. Clearinghouses 
with fewer than five research documents were omitted from further analysis • 
The scores from the remaining twelve clearingihouses were analyzed throixgh a 
one-way analysis of variance. 

To a-nswer the fourth research question (Does degree of refereeing 
make a difference?), the reports were divided into five groups on the 
"basis of the sponsorship of the research: (l) American Educational Research 
Association (AERA) papers, (2) papers from other conventions, (3) papers 
from the Educational Testing Service, {k) masters or doctoral theses, and 
(5) papers falling in none of the other categories. The scores from the 
resulting five groups were analyzed througja analysis of variance. 

To answer the fifth research question (Do Decision Makers and Research 
Specialists differ in their ratings of the quality of the research?), the 
frequency counts of the five rating scales of the Research Specialists and 
the Decision Makers were compared through Chi Sq;uare analysis. The scores 
of the two groups on these scales were also compsured hy means of a repeat- 
ed measures analysis of variance . 



33 

Two judges examined the' documents were rated "Revise" or 

"Reject" ty the reviewers to determine the tasis for this rating. These 
documents were assigned to "broad categories of reasons for rejection or 
revision on the basis of information provided in the open-ended items on 
the questionnaires. These categories for the Research Specialists and 
for the Decision Makers were then compared to determine \rtiether differ- 
ences in the reasons behind such ratings existed. 

In addition, two groups of documents were selected for more detailed 
analysis; (l) reports rated "Accept" by the Research Specialists and 
"Reject" by the Decision Makers, and (2) reports rated "Reject" by the 
Research Specialist euid "Accept" by the Decision Maker. Specific reasons 
for rejection were examined for these two groups of docvmients. 

To answer the sixth research question (Are differences in research 
sophistication of Decision Makers related to their ratings?), the 
reports were divided into those which had been rated by a Ph.D. Decision 
Maker and those which had been rated by a non-Fh.D. Decision Maker, and 
the resulting groups of reports were submitted to a repeated measures 
analysis of variance. 

In addition, the documents were divided into four groups according 
to institutional affiliation: (l) those evaluated by a Decision Maker who 
was affiliated with a major research organization or a person designated 
as a research specialist for an educational institution or system, (2) 
those evaluated by a Decision Maker affiliated with a college, university, 
or technical school, (S) those evaluated by a Decision Maker affiliated 
with a state or local education system, and (k) those evaluated by a person 
affiliated with a single elementary school, high school, or nursery school. 
These resulting groups of reports were submitted to repeated measures 
analysis of variance. 
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To answer the seventh research question, an attenrpt was made to 
divide the reports into four groups on the "basis of the recommendations 
concerning implementation suggested by the Decision Maker: ^(l) high re- 
commendation for implementation, (2) qualified recommendation for imple- 
mentation, (3) recommendation not to implement, and (k) assertion that 
no recommendation to implement or not to implement could he made on the 
"basis of the report. These classifications were derived from separate 
readings and categorizations "by two judges of the question, "What recom- 
mendations would you make with regard to the implementation contained in 
this report?" from the Information Quality and Uses Form. This attempt 
at classification was eventually found to he impractical. Consequently, 
no direct analysis was possible. However, inferences were drawn from 
other information provided by the Decision Makers in an attempt to answer 
this research question. The reasons for rejecting the attempt at class- 
ification and the basis of the inferential approach are explained in the 
Results section of this report. 

For similar reasons, a direct approach to the eighth research question 
was impossible • The procedure of drawing inferences from other inform- 
ation provided by the Decision Makers was again followed in this ana.lysis. 

To answer the ninth research question, the reports were divided into 
four groups according to the importance of the decision and the degree of 
relevant knowledge of the Decision Maker who evaluated the document: (l) 
important decision with a high degree of knowledge, (2) important decision 
with a low degree of knowledge, (3) minor decision with a high degree 
of knowledge, and (h) minor decision with a low degree of knowledge. This 
information was derived from questions 6 and 7 of the Information Quality 
and Uses Fcrm* ^ Any response falling at four or below on the 7-point 
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scale was classified as "uningportant decision" or "low knowledge • " The 
resulting four groups were analyzed through repeated measures analysis of 
* variance . 

Information relating to years of experience of the Decision Makers 
and number of publications of Research Specialists was tabulated ^ and 
means euid medians were computed. 
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RESULTS AMD DISCUBSION 

Returns of the Evaluation Forms 

Conipleted evalviation forms were obtained for 103 of the 113 documents 
(including the reliability documents) sent to Decision Makers. Completed 
forms were obtained for 111 of the lll+ docximents sent to Research Special- 
ists. Thus the percentages of return were 91*1^ for the Decision Makers 
and 97 •jfo for the Research Specialists. In cases where responses were 
totally ambiguous, or in the few cases where the wrong evalviation form 
was returned, telephd|ne follow-ups were made to obtain the desired data. 
In cases where isolated itemis were left blank, these were treated as "No 
response" in the analyses. 

Ratings of the same doctunent by both a Decision Maker and a Research 
SpeciaJList were obtained for 90 of the 102 non-reliability documents. 

Reliability and Int er c or r e lati on s of the Rating Scales 
The inter-reader reliabilities of the scales (based on the subsamrple 
of twelve documents) were low, as might he expected of 5-point and 3-point 
scales. None were significant at the .05 level. The lowest reliabilities 
were those for the Accept-Revise-Reject scales for the Research Specialists 
i sample (.15) and for the Educational Decision Makers sanrple (.24). The 

other reliabilities ranged from .27 to .56. A complete listing of the 
reliabilities is included in Table 2 of Appendix M. 

Such low inter-reader reliabilities in ratings of research reports 
is not rare. Editors of journals are often faced with high rates of dis- 
^ . agreement among their reviewers, and Wandt(l967) reported simlax low 
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reliabilities. Nuimally (1967) points out that while extremely high 
reliabilities are often essential in applied settings, these would often 
he wasteful in hasic research. In the present study, hi^er reliahilities 
could prohahly have heen obtained hy using a lengthier and nxDre detailed 
scale, such as the conrplete scale designed hy PerseU (1966). However, 
in view of the ListLted budget and the llMted amount of time reviewers 
were expected to spend in evaluating the documents, the present short 
checklists were preferred in spite of their low reliabilities. Since the 
rating scales were to be used to compare groups of documents rather than 
individual documents, the obtedned reliabilities were deemed adequate to 
determine significant differences and relationships. 

The inter correlations of the various rating scales (based on the 
full san^ile of 102 documents) are shown in Table 3 of Appendix M. For 
the Research Specialists sample, these inter correlations ranged from .kZ 
to .76, and for the Decision Makers from .55 to .81. AH inter correlations 
in both sanples were significant at the .01 level. The intercorrelations 
of the Research Specialists* ratings with the Decision Makers* ratings on 
the same scales were near zero, ranging from -.13 to .19, all non- signif- 
icant at the .05 level. 

Quality of the Documents 
The results of the rating of the documents by the Research Specialists 
are shown in Appendix M, Tables k and 5. Fprty-four percent of the reports 
were rated as either Poor or Goii[5)letely Incompetent in the overall rating 
(Scale a) . The Research Specialists recommended that 8% be accepted for 
publication, that 38^ be revised, and that the other 5*4^ be rejected. 
(Reasons given by the Research Specialists for rating a document Revise 
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or Reject are discussed in the succeeding section of this report, where 
these reasons are coDzpared to those provided "by Decision MaJcers for sim- 
ilar ratings.) Regarding the theoretical contribution of the document 
to the discipline to which it pertained (Scale C) , l6% were rated as 
either Above Average or Excellent, whereas 56^ were rated as Below Average 
or Inconrpetent . The practical contribution of the document to educational 
practice (Scale D) was rated as Above Average or Excellent in 2k% of the 
cases , and in 4l^ of the cases as either Below Average or Incompetent . 
Use of research methodology (ScaJLe E), was rated as Above Average or Excel- 
lent in 2ki) of the reports, but as Below Average or Incompetent in 51^ 
of the reports. 

The qusuLity of these documents can be corapexed on the basis of the 
Accept-Revise-Reject (ARR) category to the quality of the doc\iments in 
Wandt^s (1967) study of published educational research. Such a comparison 
(Table 6, Appendix M) indicates that the documents in the present study 
are rated as poorer in this category at the .05 level of significance. 
Several factors should be considered in evaluating such a comparison. 
First, the reviewers of the documents in the present study were drawn as 
far as possible from the same set of reviewers used -in the Wandt study. 
Thus there is a basis for comparison. Second, as will be seen later in 
the present study, the ARR category was one of the most difficult areas 
in which to demonstrate significant differences. This was largely because 
of the small variance of the three point scale. Third, Wandt 's study was 
based on articles published in I962, whereas the documents in the present 
study were written between 19^9 and 1971. This weakens the comparability 
somewhat. If it can be shown that the quality of published educational 
research literature has improved since 1962, the present analysis may 
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reflect a conservative estimate of the differences in qiiality between 
current published research and the research in RIE * 

These ratings are supported by a high degree of authority. Seventy- 
six percent of the Research Specialists had taught courses in research 
laethodology* The tabulation of their ntonber of professional publications 
indicated a mean of 39*9 publications and a median of 30. It should be 
pointed out that this estimate of the number of publications is conservative: 
authors \dio listed "at least 30" were recorded as having 30 publications, 
and anyone listing over a hiondred publications (Some had over 150^ was 
recorded as having 99 publications for simplicity in couiputer coding. 
Thus any allegation that the Research Specialists were naive or out of 
touch with current educational trends would be hard to support. 

Research Specialists vs . Decision Makers 
Tables k and 7> Appendix M , conrpare the ratings of the Educational 
Decision Makers to those of the Educational. Research Specialists* On clII 
scales, the Decision Makers showed significaJit tendencies to rate the 
reports more favorably than did the Research Specialists. For example, 
the Research Specialists rated only 5 reports as "Excellent— A model of 
good practice," whereas the Decision Makers rated 15 reports in this cat- 
egory. Similarly, while the Research Specialists rated 10 documents as 
"Con5)letely Incompetent," the Decision Makers rated only 1 document in 
this category. Likewise, the Research Specieilists rated only 8 reports 
"Accept" and 52 reports "Reject," ^Aierecus the Decision Makers rated 30 
"Accept" and only 23 "Reject.'' The some trend was evident on all rating 
scales. 
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Table ik, Appendix M, coaipares the reasons given for "Revise" and 
"Reject" ratings by the Research Specialists and Decision Makers. In 
general, the Research Specialists rejected documents on the basis of poor 
design or educational triviality, whereas they recommended revision when 
the design was unclear, \rtien the report was too long, when there was 
question about the representativeness of the saurple, and when the analysis 
appeared to be incomplete. Thus there appeared to be a clear distinction 
between the reasons for revision and rejection: "Revise" was recommended 
when shortcomings appeaxed which could be corrected by rewriting the repoi*t 
or parts of it, and "Reject" was recommended when the entire study would 
have to be redone. It is noteworthy that these reasons for revision 
indicate the same benefits which psychologists have reported as a result 
of submitting rough drafts of their research papers to peers prior to 

* 

formal publication (Garvey & Griffith, 1971). 

The dichotomy between reasons for revision and reject:!, i;?!! vrcts not as 
clear for the Decision Makers . Poor design was given with about equal 
frequency as a reason both for rejection and for revision. Extreme com- 
plexity or technicality, a reason for revision which did not appear among 
the Research Specialists, was fi?equently cited by the Decision Makers. 
In four cases the Decision Makers recommended that the report be revised, 
even though they made no adverse comments about it. The recommendation 
that the stxxdy be replicated with the reviewer *s "pet theory" included as 
part of the study was unique to the Decision Makers. It would seem, then, 
that the reasons given by the Decision Makers for revision and rejection 
were somewhat different from those of the Research Specialists. When they 
recommended "Revise," this apparently sometimes meant that the entire 
study, or major parts of it, should be repeated with different procedures. 
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This meaaiing of ''Revise*' was not apparent among the Research Specialists. 
In addition, many of the Decision Makers* reasons for revision focused on 
making the report more meaningful euid useful in practical situations. 

jFurther analysis indicated that there was only one document rated 
"Reject" by the Decision Maker but "Accept" by the Research Specialist. 
This document was labeled "extremely esoteric" aoid "excessively technical" 
by the Decision Maker (who was a university professor) . It is notewrthy 
that the Research Specialist indicated that it should be rated "Accept" 
ft'om the standpoint of the state of art in the discipline to which it 
applied, but by best educational research standards it should be rated 
Revise or possibly Reject. 

On the other hand, eleven documents received an "Accept" rating, from 
the Decision Maker and a "Reject" fjrom the Research Specialist. An exam- 
ination of the reasons given for rejecting these documents indicated that 
one of these documents was rejected primarily because it was too lengthy 
and contained relatively minor problems in sampling techniques . Five 
documents were rejected because of serious errors in research design, and 
three because of the extreme triviality of the research. Two others were 
rejected because of a combination of poor design ajid triviality of the 
research. 

One Research Specialist indicated that althou^ the report attempted 
to measure change, no measure was taken of the dependent variable prior 
to the introduction of the experimental variable, and no control group was 
employed. Another Research Specialist indicated that "...what they report 
is in the nature of a series of mediationol checks to determine whether 
an independent variable has been manipulated. But the question of whether 
that independent variable has effects upon teaching performances is not 
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even addressed." Another stated, "The research adds nothing to our know- 
ledge. It is only tusy work that should never have teen funded. The 
report would not pass as a Masters thesis. Indeed, it wotild hardly pass 
as a term paper in view of the failure of the author to show a familiarity 
with the important work that has been undertaken in related areas." Such 
analysis and anecdotal information sxiggests that these papers which were 
rated "Accept" hy the Decision Makers were rejected hy the Research 
Specialists for more than minor flaws. In eleven out of the twelve cases, 
the concern was with extremely poor methodology or with the extreme 
triviality of the research, rather than with problems t*hich would he 
considered a matter of taste . 

These analyses indicate not only that the research reports in this 
study are of low quality, hut also that Educational Decision Makers in 
general do not recognize this inferior quality. The Decision Makers show- 
ed consistent tendencies to overrate the quality of the reports they re- 
viewed, and in many cases they rated 8ls high qixality reports with ex- 
tremely serious Inaccuracies. If this is true, then it would seem that 
ERICAS Research in Education has fallen somewhat short in its crusade to 
"sagaciously clear only the best for inclusion in HIE" (ERIC> 1970a), 
Quite the contrary, the evidence indicates that rather than providing 
high quality information, RIB often disseminates vast quantities of low 
quality information, ^ich is very likely to be regarded as high quality 
by Decision Makers. 

Raxige in SoTahistication of Decision Makers 
An examination of the positions of the Educational Decision* Makers 
(Appendix C) indicates that a wide diversity exists in the qualifications 
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of the members of this group. It is evident that a nmber of • these Decision 
Makers could legitimately he classified as Research SpeciaJists • Although 
the intent of the present study was to submit the documents to a group of 
"typical" Educational Decision Makers, it is evident that this is not what 
actually took place. Since the Decision Makers in this study were volun- 
teers, it is possible that only persons with a certain degree of confidence 
in their ability to read and evaluate research reports would have respond- 
ed affirmatively to the introductory letter (Appendix B) . In addition, 
majoy of the RIE documents dealt with higher education or teacher education, 
and in such cases the most appropriate Decision Makers were college or 
university personnel. SiAch Decision Makers are more likely to have some 
research orientation. These two biases also seemed to interact: college 
level personnel with research interests were more likely to volunteer to 
take part in a study of research quality. 

Thus it seems possible that many of the Decision Makers were among 
the elite of this group. When an educational decision is to be made in a 
"typical" setting, it hardly seems likely that it vrill always be referred 
to a Decision Maker as qualified as these • 

The bias of this Decision Maker sample is toward the more critical 
end of the scale; for example, more tendency to reject, etc. 'fhus the 
differences found in this sample would probably be markedly greater among 
more typical Decision Makers. 

Table 8 and Figures 1 through 5 in Apperi&Lx M reflect an attextrpt to 
explore this line of reasoning further by analyzing the ratings of the 
documents by Decision Makers with different degrees of research expertise . 
It was hypothesized that holdet*s of doctoral degrees would have a higher 
level of research sophistication than Decision Makers without doctorates . 
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The docionents were divided into those reviewed by a Decision Maker with a 
doptorate and those reviewed by a non-doctoral Decision Maker. The ratings 
of the Research Specialists suxd the Decision Makers were then con^ared. 
If the above hypothesiE were true, a significant group by trial inter- 
action should appear. Table 8 indicates that a significant interaction 
was in the expected direction on the rating of practical usefulness 
(variable k) . Nonsignificant trends appeared on three of the other four 
variables (variables 1, 3? snd 5). Thus, Table 8 provides some evidence 
that less sophisticated Decision Makers are more likely than sophisticated 
Decision Makers to over-rate the quality of a reseaurch report. 

There Bxe perhaps several factors which prevented the emergence of 
more significant results. First, it is probably naive to assume that a 
doctoral degree automatically implies a high degree of research sophistica- 
tion. Second, a nuniber of the Decision Makers listed as non-doctoral 
reviewers were at the time persuing doctorates on a part-time basis. In 
a few cases, the Decision Maker was a full-time Ph.D. candidate working 
with an educational system. Such non-doctoral Decision Makers would 
probably be in even closer touch with research methods and trends than 
would zaany of the doctoral Decision Makers. It seems possible, therefore, 
that the analysis summarized in Table 8 is not based upon as clear-cut a 
dichotomy of research sophistication as would be desirable, (in both this 
and the siibsequent analysis, moreover, the low reliabilities of the 
scales would operate against finding significant differences.) 

Table 9 and Figures 6 through 10, Appendix M, reflect a second attempt 
to analyze the effects of research sophistication on the ability to make 
an accurate detennination of the quality of reseaxch. It was hypothetiized 



that a continuum of levels of research sophistication exists among the 
following four groups of Decision Makers: (l) memhers of research and 
evaluation organizations, (2) members of college faculties, (3) state and 
local school system personnel, and (4) faculty members of individual 
schools within such systems . This breakdown revealed significant inter- 
actions in the expected direction in two cases (variables 3 and k) and 
non-significant treiids in the other three cases. * 

These aimlyses provide evidence that Decision Makers with stronger 
resesLTch orientations are more likely to evaluate research in a manner 
which will correspond to the evaluations made by Research Specialists. 
However, the research dissemination needs of these research and evaluation 
personnel ajid of the college faculties are already well served by journals 
and other means of standard research and professional communication. On 
the other hand, the needs of the Decision Makers at the state, local, and 
individual school levels (the non-sophisticated end of the continuum) are 
not as ewlequately served by such standard professional dissemination 
procedures. These levels of Decision Makers predominate in our educational 
systems and are presumably a prime target audience for ERIC's RIE services- 
It is these Decision Makers who are likely to evaluate research reports 
most divergently from the evaluations of Research Specialists. 

It may seem that these analyses belabor the obvious. It is not 
revolutionary to discover that persons who are more sophisticated in 
research make more sophisticated eval\aations of research. However, this 
analysis does have important in5)li cations. Stuff lebeam (l971> p* 63) has 
demonstrated that evaluation must be most extensive and most accurate 
when there is little information avaiJable to the practitioner who must 
make a decision or when the decision maker cannot use available informa- 
tion in its present form. If this is true, then it is not the personnel 



k6 

at the sophisticated end of the research sophistication continuiam who 
most need information by which they can evaluate accurately, hut rather 
the non-sophisticated practitioners who have need of a great deal of 
information which they must accept or presume to he accurate. A useful 
dissemination system would he one which presents such decision-making 
practitioners with infcirmation which they recognize as pertinent and ac- 
curate eind which is indeed of high factual quality. A dissemination sys- 
tem which cannot provide such a guarantee of quality to the Decision 
Makers at the lower end of this continuum would apparently "be a disservice 
rather thaui a useful assistance. 

Relation of Findings to Decision Making Processes 
Originally it was intended to group the responses of the Decision 
Makers into varying degrees of willingness to implement the research find- 
ings of their report according to their responses to question eight on 
their evaluation form (What recommendation would you make with regard to 
the implementation of the information contained in this report?). In 
many cases the Decision Makers provided responses which could he appro- 
priately classified. In other cases, however, the Decision Makers respond- 
ed hy specifying the kinds of further research they would like to see or 
revisions which could be made in the documents. Sometimes "no impleraen- 
tation recommended" obviously meant that the information contained in the 
document was useless; at other times it meant that the information was 
merely inapplicable to the Decision Maker's current situation; and at 
other times it was difficult to discern what was meant. Because of such 
diversity in approaches by the Decision Makers to this question and be- 
cause of the ambiguity of many of the responses, the planned analysis 
based on this question was not implemented. Consequently, the results 
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analyzed in this report are confined to the perceptions of the Decision 
Makers towards the quality of the documents, rather than with hypothetical 
decisions to implement or not to iisplement the results of a given report. 

The present study, therefore, examines the actual decision making 
process only to the extent that it can legitimately he assumed that a 
Decision Maker, given the opportunity sind need, is likely to ing)leffient 
research which he views as high quality and to decline to implement 
research which he views to he low quality. This assumption appears to he 
valid. 

Variations in Degree of Quality Control of the Reports 
Although it would be unwise to generalize on the hasis of the very 
small sanjJles from the various clearinghouses in the present study, 
Tahle 10 of Appendix M suggests that differences in quality of documents 
are not related to the ERIC clearinghouses from which they originated. 
Thus no differences in quality control among the clearinghouses can he 
hypothesized ftrom the results of this study. 

Within RIB itself, however there may be scaae general differences 
in the degree of quality control ^ich a document might receive. For ex- 
ample, documents from research-oriented conventions are usually somei*at 
refereed before acceptance and subsequent submission to ERIC. Such docu- 
ments certainly receive a greater degree of quality control than a final 
report which is sent to a funding agency and is almost simultaneously in- 
corporated into iSIC's RIE system. 

Table 11 €uxalyzes an arbitrary arrangement of the documents in the 
present study according to approximations of the degree of quality control 
before the documents are incorporated into ERIC. The results indicate 
that on Scales A, C, and E there ajrpear to be differences in quality. A 
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rank-ordering of the means on these three scales indicates that non-spon- 
sored reports and papers from minor conventions are consistently rated 
lowest in quality, while the more tightly controlled Educational Testing 
Service (ETS) papers, the Ph.D. and Masters theses, and AERA papers are 
rated higher. 

The degree of quality control or refereeing would seem to "be a factor 
in these differences. This factor would perhaps "be more evident if it 
could be more adequately isolated. For exaniple, the "Other" category un- 
doubtedly contained some papers which were submitted to large amounts of 
refereeing which could not be detected in the present study. In addition, 
conventions, such as that of the AERA, are viewed with varying degrees 
of formality and informality (cf . Garvey & Griffith, 1971) by different 
contributors and referees connected with these conventions. Thus quality 
control probably varies widely within the groups in Table 11. However, 
the present evidence does indicate that considerable differences do in 
fact exist between the quality control evidently exercised by ETS, for 
example, and the perhaps more haphazard acceptance policies of non-research- 
oriented conventions and meetings. 

The Decision Setting 
Table 12 of Appendix M and Figures 11 through l6 indicate that sig- 
nificant differences in the amount of divergence between Decision Makers 
and Research Specialists were related to the decision setting. The great- 
est discrepancy consistently appears in the High Information Grasp—Major 
Decision setting. Decisions involving major changes are viewed by Stuf- 
flebeam, et al. (l97l) as necessitating the most intense and most accurate 
evaluation. It must be noted, however, that the same shortcoming which 
applied to the attempts to clasfiify the Decision Makers* plans to 



^9 

implement the research findings apply here. Both information grasp and 
the magnitude of the change were based on the perceptions of the Decision 
Maker himself. As was indicated previously, the Decision Makers apparent- 
ly approached the problem of implementing change from several different 
points of view, and consequently results based on self -ratings into these 
categories are somewhat tenuous. 

Summary of Results and Discussion 

Overall, only 33^ of the ERIC documents are backed up by enrpirical 
data. (This data is summarized in Table 13) . It is certainly possible 
that some documents could be useful without being siipported by empirical 
data; for example, a good bibliography, a review of the literature, or a 
discussion of a theory or educational practice could all be useful 
without empirical data supported by statistical analysis. However, such 
reports must be viewed primarily as appeals to authority in the absence 
of supporting data, and thus the fact that 6?^ of the material disseminated 
through RIE is without accompanying en?)irical support is in itself perhaps 
cause for concern. It is noteworthy that these figures correspond closely 
to those in the previous Asher and Vockell (l97l) pilot study. 

The present study, however, has been concerned with the quality of 
those reports disseminated through RIE which do contain euipirical data 
The evidence presented in this study indicates that, while some high 
quality reports exist in RIE 3 the reports cure on the whole of low q\iality. 
If the quality of the research-oriented documents in RIB is representa- 
tive of the overall quality of RIE documents, this would mean that only 
32.5^ of the RIE documents would be rated as "good" or "excellent" quality. 

In addition to pointing out the apparent low quality of the research- 
oriented documents in RIE 3 this study has provided evidence that this low 
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quality is in many cases not recognized ty the educational practitioner 
who must make decisions "based on the infoimtion vhich he receives. As- 
suming that these Educational decision Makers make decisions "based on in- 
formation which they receive (and it is only to the extent that this as- 
sumption is true that the information is worth disseminating in the first 
place), putting low quality information in their hands while they have 
the impression that it is actually Mgh quality information is likely to 
lead to implementation in education systems of faulty research. 

On the one hand it is apparent that the classification of a report 
as "excellent" or "incompetent" is a somewhat tenuous endeavor. The 
purpose of the project on which the report is "based, the state of techno- 
logy in that domain of work, the amoxmt of money available to the researcher 
or evaluator, the amount of time available for research and/or evaluation, 
the people badgering the researcher for information — these and many other 
factors need to be taken into consideration (cf . Runkel & McGrath, 1972, 
pp. 220 eind 427). An evaluation of a project may be useful to a limited 
audience, even if it is haphazardly performed, simply because a haphazai'd 
evaluation may convey at least some new information which they may not 
have had access to otherwise. Such an evaluation may even be informally 
communicated to others. But, on the other hand, the transition from a 
r'elatively informal to a relatively formal mode of communication has been 
shown to give an misleading inipression of authenticity to such information 
(GarveySc Griffith, 1971). 

The purpose of RIE is to supply rapid, widespread dissemination of 
educational information. A useful avenue of research would be to deter- 
mine what savings of time there actually is in this relatively unrefereed 
process. A relatively slight savings in time would not really be worth 
a considerable loss in scientific accuracy. 
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CONCLUBIONS AND RECOMMENDATIONS 

Effects of Rapid Dissemination 
ERICAS RIE provides a considerable savings in time required to dis- 
seminate educational, research information. These gains in speedy however, 
may he accompanied hy several undesirable side effects: 

(1) The possibility of useful feedback from prepuhlication exchanges 
with peers is minimized or eliminated. Garvey and Griffith (l97l) have 
shown that such prepuhlication exchanges often result in significajit 
modifications in the final version of a manuscript, such as reanalysis 

of data and redefinition of concepts. Such inqprovements would make the 
research much more useful for educational practitioners, hut they cannot 
he expected to occur when it is the prepuhlication version itself which 
is being disseminated. 

(2) The distinction between formal and informal communication is 
clouded. Garvey ajid Griffith (l97l) have emphasized the iniportance of an 
informal forum where researchers and authors can present half -formed ideas 
for critical appraisal and commentary from their peers before re-evaluating 
them and presenting them to the public. Even if such information is 
somewhat inaccurate it does little harm if it is confined to this Informal 
channel. If such information is disseminated directly to the public, 
there is evidence that it may lose its identification as informal and be 
viewed .by the public as definitive and sound information. 
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(3) Inaccurate or low quality information may be disseminated. This 
is likely to result from the absence of prepuhlication scrutiny which norm- 
ally accompanies the refereeing of articles submitted for journal publi- 
cation. The present study has demonstrated that such low quality of re- 
search information is often the case with RIE documents. 

(h) The dissemination of large quantities of low quality inform- 
ation in am information channel tends to obfxiscate the impact of the 
high quality information disseminated in the same channel. A comment 
such as "lt*s only on ERIC microfiche. It*s probably no good anyway," 
would tend to bias interested decision makers against the good documents 
which contain high quality research in the same system. 

(5) The probability of postpublication scrutiny in the form of crit- 
ical comments, critiques, rejoinders, and letters to the ERIC Clearing- 
house is lowered. In theory, a critique of an RIB document would be ac- 
cepted ajid disseminated as rapidly as the original document. Research by 
Vockell and Asher (1972), however, suggests that active solicitation as a 
matter of journal policy and practice is required to obtain such critical 
comments. The fourteen hundred docvmients reviewed in the present study 
and in the Asher and Vockell (1970) pilot study included not a single 
examrple of a critique of another RIE document. "The absence of such post- 
publication scrutiny eliminates the possibility that a reader who read an 
erroneous document will later also read a subsequent document where the 
errors are described and corrected. 

(6) The reader has no assurance that the research reported is ac- 
ciarate, of high quality ^ and related to previous research in the same 
area. This role of quality control is traditionally allocated to the 
editors or board of reviewers of a journal. The ill effects resulting 
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from the absence of refereeing eire minimized to the extent that the reader 
can evaliiate the quality of the docianent himself. However, the present 
stiidy provides strong evidence that Educational Decision Maimers are in 
many cases unahle to make this evaluation. Consequently, there is con- 
siderable likelihood that inaccurate, low quality information will be ac- 
cepted as accurate, high quality information by practitioners who must 
make decisions based on the research available ^to them. 

The advantages of a rapid dissemination system such as R3E , there- 
fore, must be weighed against these imdesirable side effects. A small 
savings in time is perhaps not worth the risk of encountering these side 
effects. They can perhaps be reduced by improving and upgrading the 
system. As it is now, the system might be doing more harm tlian good for 
itk users . 

Rapid Dissemination with Quality Control 

An ^ef f iciext course of action would be to provide maximum savings in 
time, while simultaneously eliminating the undesirable side effects of 
rapid dissemination. This would involve changing the RIE system in such 
a way as to retain as far as possible its rapid dissemination aspects, 
while also introducing forms of quality control. 

The most obvious form of quality control would be prepublication 
screening: referee the reports and sinrply decline to disseminate low 
quality information. High quality documents would be included in RIE im- 
mediately. Other documents would be either rejectiad or returned to the 
authors for revision, .This refereeing could be accomplished either by 
employing research specialists with each clearinghouse for this purpose, 
or by sending the documents to outside referees and basing pa^rment on 
meeting a deadline. 
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Such refereeing would not, in itself, cause undue delays in pub- 
lication « The time consiaming aspect of most professional publication is 
not the refereeing process, hut rather the space limitation in the journal, 
which results in a "publication lag." Pay journals are often able to 
overcome the publication lag by having the authors pay the costs of print- 
ing and adding additional pages as necessary. The American Psychological 
Association journals have a practice of giving the authors the option of 
paying the ex±ra cost involved in having an article published in an earlier 
issue of the journal than would be permitted by the publication lag. All 
other things being equal, refereeing without the publication lag which 
accompanies page limitations would involve a relatively minor delay. 

A source of delay which often accompanies the refereeing process is 
the revision process. For example, in its initial form a document may 
contain inadeqtxate descriptions of an experimental progreun, or the subjects 
involved in the study, or of the experimental controls. Or perhaps axi 
inappropriate analysis was used. The correction of these inadequacies 
would involve either a rewriting of the report or performing additional 
analysis. 'Such steps would certainly be time consuming and would slow 
down the speed of dissemination, but it seems obvious that the resulting 
improvements in quality would make the research more useful to practi- 
tioners who would want to use it as a basis for their decision making. 

It is also possible, of course, that closer examination and reanaly- 
sis would indicate that the document was inaccurate and is not worth 
reviewing. In this case it could hardly be argued that such a discovery 
prior to dissemination would be time constuning or otherwise detrimental 
to the dissemination process. The refereeing process in this case would 
serve a useful Elimination function. 
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Such a refereeing process would be most useful to the extent that it 
is rapid. Likewise, the revision process would he most useful to the 
extent that it is performed rapidly. Rather than merely returning a 
document with the assertion that it "might he accepted if revised ad- 
equately," many professional journals are adopting a policy of suggesting 
specific revisions. In other cases the reviewer or editor actually writes 
a tentative revision auid returns this to the authors for approval and 
further revision prior to publication. Such procedures expedite the 
review process auid cotild he applied to the RIB review, revision, and dis- 
semination process. 

It seems possible that persons who submit reports for dissemination 
in RIE might differ from persons who submit articles to professional 
journals in an important respect: journal authors might be to an extent 
motivated by the emphasis placed on publications as a part of the univer- 
sity promotion process, whereas RIE reports carry less weight in this 
area. Consequently, authors of RIE reports might be less motivated than 
authors of journal articles to. revise and resubmit a manuscript. This 
could result in the loss of many desirable manuscripts which would other- 
wise appear in RIE . Including suggested revisions as a part of the refer- 
eeing process could alleviate this problem. If all the author has to do 
is a.dd some specifically requested information and retype paxts of the 
document, this might not be viewed as a serious burden. When incentives 
to encourage the original author to revise the document do not exist, in 
some cases it may even be desirable to have the ERIC clearinghouse revise 
the document. Precise roles and incentives for the author and the review- 
er would have to be refined, but it does seem that the problem is sur- 
mountable , 
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Discussion thus far has focused on prepublication refereeing. Post- 
publication comments eind interactions have "been shovn to he essential, even 
in professional journals. A possible application to RIE might he to en- 
courage such interaction hy giving priority to very short critiques which 
point out shortcomings in other RIE documents. Such critiques or rejoin- 
ders could he submitted to immediate, high priority review, and quick dis- 
semination could be guaranteed after a factually accurate critique is sub- 
mitted . 

The application of an efficient refereeing process to the RIE dis- 
semination system would deal with the major problems of ERIC emerging from 
the present study. Such a refereeing process would be similar to the 
process which accompanies publication in professional joiarnals. The basic 
premise would not be that only the best examples of professional research 
would be accepted, but rather . that all high quality research would be dis- 
seminated as soon as this quality could be established. Moreover, length 
of the article, which is a major consideration in page-conscious profes- 
sional journaJLs, would not be a major concern. Lengthy appendices and 
copies of measurement instruments could be readily included. This is one 
of the major and unique advantages of RIE . When appropriate, critical 
comments of these instruments could be added by the author or by the lUE 
reviewer. Since the refereeing process would be unique to RIE , contin- 
uous experimentation, evaluation, and refinement would be necessary. -This 
refereeing and revision process is merely a suggestion. In view of the 
concern about RIE indicated in the present study, however, such suggestions 
would iseem to provide a useful starting point . 

The present report has concerned itself with the research-oriented 
aspects of RIE . Eighty percent of the documents in RIB contain no 
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empirical research, and in mny cases this absence of empirical data is 
entirely appropriate. Although it would be rather arbitrary to assume 
that the q^iaJLity of this non-research literatvire is of similatrly low 
quality, it would nevertheless appear to be safe to infer that variations 
are present in the quality of this literature. To the extent that such 
variations between high and low quality occur and to the extent that 
Educational Decision Makers are xinlikely to recognize low quality without 
assistance, the refbreeing, revision, and elimination procedures recom- 
mended for the research literature in RIB would also apply to the non- 
research literature. 

Areas for Further Research 
The findings summarized above suggest several questions for subsequent 
research: 

(1) How much of a publication lag currently exists within the RIB - 
system?.; > RIB dopumentslare a.poompanled ^by.a. "puVlieation date" which oindi- 
cates when, the document was actually written. A comparison of these 
dates with the act"ual date the document is indexed and made available 
through RIB indicates that the median age of a document is about 6-12 
iaonths. Many of the documents which appear in RIB may subsequently ap- 
pear in professional journals. Useful reseai*ch could be conducted to 
determine how much longer it takes to publish ah article in a professional 
journal than to release it through RIB . 

(2) When the same research is reported in both RIB euid in a profes- 
sional journal, how do the two reports compare in terms of quality? What 
changes come about as the result of the refereeing the documents undergo 
when submitted for professional publication? 



(3) How does the quality of RIB docmnents conrpaxe to the qixality of 
currently published research journals? Wandt's (1965) study reviewed I962 
journal publications, and it seems likely that the quality. of the refereed 
literature may have changed since that. date. 

(h) What is the quality of the non-research literature in RIE ? In 
melny cases, the absence of enrpirical data is entirely appropriate; and in 
such cases the problem is to evaluate this non-research literature by ap- 
propriate criteria. In other cases, the information is relatively mean- 
ingless or trivial without supporting data. In some cases the authors of 
the documents apparently believe that they have data when in reality none 
is present. 

(5) Can RIE documents be refereed without a disproportionate loss 

of time in the dissemination process? Although refer eeing of professional 
journals has been shown to have its. own shortcomings, the results of the 
present study indicate that refereed literature is better than the non- 
refereed literature. If refereeing can be done quickly and can bring 
about a degree of quality control, then it would certainly be worth a 
brief delay which would be involved. The suggestions provided in the 
previous section would provide a useful starting point for such research. 

(6) Can a useful sjrstem of postpiibli cation exchange and scrutiny be 
introduced into RIE ? What would be the effects of active solicitation 
emd rapid dissemination of carefully written critiques, critical comments, 
and rejoinders concerned with information disseminated in RIE (or even 
elsewhere)? 



Conclujsion 

This study has shown that the resestrcsh disseminated throiigh Rffi is 
often of low quality and that this low quality is often xmre cognized by 
the Decision MaJcers who rejxd the research and "base decisions upon it. 
Moreover, it has been shown that Decision Makers who lack research soph- 
istication are most likely to overrate the quality of the reports. It 
hajs been argued that such a result is not only a waste of time and moneya 
but rather am active disservice to the Decision Makers. 

However, this study has also shown that there are wide variations 
in the quality of the documents within RIE , Quality in dociunents 
seems to be related to the degree of refereeing or quality control prior 
to its dissemination. Therefore, it would seem useful to introduce a 
system of rapid refereeing into the RIB system. The implementation of 
such a system would retain the advantages of RIE 'e rapid dissemination 
system, while reducing its negative side effects. 
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I APEEHDIX A 

THE DOCIMENTS IN THE STUDY 

.^ ^ ED 0k2 087 Influences of the Kentucky cooperative extension home economics 
^ program. Part I 

H u EI) 0l|2 lk9 Influence of Initial and final consonants on vowel duration 
in CVC syllables. 

C ED 0k2 213 Proprioceptive factors in operative timing. 

^ ^ ED '0k2 287 A 3-year study of continuity of creative growth under a 
cognitive-structured approach to educational stimulation. 

' ED 0k2 363 Optimal use of a computer "based instruction system in an 
existing urban school district. Final Report. 

" ' ED 0k2 kl3 Report on the subcommittee of the status of academic women 
on the Berkeley campus. 

1 - ED 0k2 k99 The modification of teacher behaviors which modify child 
behaviors. Progress Report. 

The "Tell-and-Find Picture Games" for young children. 

Language survey of entering primary students (rural schools) 

Transfer from word comyponents to words and vice versa - 
beginning reading. 

Pupil mo'tivation: A rewarding experience. 

A look at combination class effects at Emerson Elementary 
School. 

Evaluation of a program for developing creative thinking in 
teachers and children at the fourth and fifth grade levels. 

National assessment of educational progress. Summary of 
Report 1. Science: National Results, July 1970. 

Some characteristics of finalists in the I966 national 
achievement scholarship program. 

Attitude change of freshman college students toward their 
role as raters of teacher behavior. 

Staffing, equipping, and operating public school reading 
clinics. 

Blueprint for a successful paraprofessional tutorial program. 
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ED Ok3 587 Prepetring teachers to involve parents in children's learning. 
Project Report. 

ED Ok3 599 Evaluation report of cooperative urtan teacher education 
program. Fall 1969? Working Paper, Vo. 3 No. 1. 

ED Okh 063 The MIA Foreign Language Proficiency Tests for Teachers: 
Advanced Students; Analysis of the performance of native 
speakers; Con^parison with that of NDEA Summer Institute 
Participants. 

ED Okh 087 A survey of rajik, salary, promotion, and tenure policies in 
15 colleges and universities. 

ED Ohk 099 Student and instructor expectations of commimity college 
instructors. 

ED okh 387 Educational innovation; Research and evaluation techniques. 
(Rep. of 1967 Suiamer Institutes) 

ED Okk hS9 Competencies needed hy seed production and distribution 
company employees . 

ED okk 6k9 Evaluation of adult hasic education workshop in western 
Tennessee, Dec. l^^-, i960. 

ED okh 7k9 Counselor hias and the female occupational role. 

ED okh 863 The effects of a program of learning gajMs upon selected 
academic abilities in children with learning difficulties. 

ED okh 963 EEDA Institute in German for cooperating teachers and teacher 
trainers. Final Report. 

ED Ok^ Ok9 Student personality characteristics and optimal college 
learning conditions. Final Report. 

ED 0I15 k63 Evaluation report: Project Insight 1969-1970. 

ED Ok^ 763 A research project of the North Carolina Advancement School. 
Simmer Session 1968. 

ED Ok6 Ok9 Bases of classification of geometric concepts used hy children 
of varying characteristics. Report from the Project on 
Situation Variables and Efficiency of Concept Learning. Part I. 

ED OkS lk9 A follow-up study of graduates from a vocational rehabilitation 
program in a residential training center for the mentally 
retarded . 

ED 0k6 2k9 Aptitude-treatment interactions in two studies of learning 
from lecture inetruction. 



67 



ED 0k6 499 Piaget*s concept of classification: A comparative study of 
, socially disadvantaged and middle-class young children. 

ED 0k6 513 Developmental groupings of pre-school children. 

ED 0k6 563 Oregon state migrant education program. Annual Project 
Report and Evaluation 1967-68. 

ED 0k6 599 An inter*disciplinary evaluation of a summer program for the 
rural disadvantaged youth in Nelson County, Virginia. Final 
Report . 

ED Clk6 987 Levels of educational performance and related factors in 
Michigan. 

ED 0k6 999 Clinical-diagnostic use of the WPPSI in predicting learning 
disabilities in grade one. 

ED 047 099 Teaching beliefs and classroom methodology of selected project 
and cooperative plan distributive education teacher coordinators 

ED Okl 113 Follow-up of 1965 graduates of Wisconsin Schools of Vocational, 
Technical, and Adult Education. 

ED 0U7 hl3 Prep€uring communication consultants for a school district: 
A study of some effects of laboratory training. 

ED 014-7 ^7 Mathematics curriculum innovation and evaluation for educable 
mentally retarded. Final Report. 

ED Olf7 513 The interaction of associative memory and general reasoning 
with availability and complexity of examples in a computer- 
assisted instruction task. 

ED OI1.7 699 Reaching the unreached; report of the regional action workshop 
devoted to the "non-user" of library services. 

ED 047 899 Predictive values of selected auditory perceptual factors in 
relation to measured first-grade reading achievement. 

ED Ok7 987 The Oakland County mathematics project. 

ED 0k8 099 Pupil control idealogy and teacher influence in the classroom. 

ED OhQ 3^9 Assessment of teachers* attitudes toward an individualized 
approach to reading instruction. 

ED OkQ I487 Testing of the guidance program, project able; development 
and evaluation of an experimental curriculum for the New' 
Quincy Massachusetts Vocational Technical School. 



ED 


Ok8 




ED 


ok8 


613 


ED 




799 



68 



Models of diffusing knowledge to commmity college faculty- 

Client motivation and rehabilitation coxmseling outcome* 

Auditory discrimiriation and Identification in foreign language 
learning, 

ED Ok& 863 Cost analysis: First step toward differentiated funding • 

ED 0k& 899 Study and comparison of the indexing of the "bibliography of 
agriculture" in relation to the indexing of 15 other 
" secondary services. 

ED Ol^9 287 An investigation of selected procedures for the development 
and evaluation of hierarchical curriculum structures, 

ED Ol^9 313 Using the VMPl to predict teacher behavior. 

ED 01^9 387 Development of educational programs for new careers in 
recreation services for the disabled • 

ED 6k9 kk9 Inventory of continuing education activities in Pennsylvania 
institutes of higher learning for the academic year 1969-7O- 
Final Report. 

ED 0k9 713 Encouraging convergence of opinion, through the use of the 

Delphi Technique, in the process of identifying an institutional 
goal « 

ED Ol^9 813 Correlation of paired-associate performance with school 
achievement as a function of task and sample variation. 

ED 050 013 The effects of perceived teacher attitudes on student 
achievement, ^ ^ 

ED 050 087 Effects of commonly known meanings on determining obscure 
meanings of multiple-meaning words in context. Report for 
the project on individually guided elementary language arts. 

ED 050 ll^9 The mini rat: Its development and some evidence of its validity. 

ED 050 263 SIM (service unit management): An organizational approach to 
improved patient care . 

ED 050 399 Manifest characteristics of interactive sequencing in the 
classroom. ' 

ED 050 k63 Organizational climate: Another dimension to the process of 
innovation. 

ED 050 663 A If-year survey of DePaul University's School of Education 
master degree graduates for I966 thru 1969- First Interim 
Report. 
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ED 050 487 Evaluation and selection of academic interns: I967-68. 

ED 050 863 The nongraded-trimester -mini course concept: Report to date. 

ED 051 0^9 A study of the measured readability level of selected 
intermediate grade social studies textbooks. 

ED 051 087 An^evaiuatiuheof the program "supportive training for 

inexperienced and new teachers" (STIHT) in New York City 
schools. 

ED 051 099 The Stanford secondary teacher education program 1959-69: 
A preliminary analysis of g^ad^xate career patterns. 

ED 051 113 Human interaction: A behavioral objectives curriculum in 
hiaman relations. 

ED 051 2k9 Relationship between teacher* vocabulary usage and the 

vocabulary of kindergarten and first grade students. Final 
Report. 

ED 051 7^9 Eic transfep^udent: An institutional variable in the 
analysis or jiiriibr year college marks. 




ED 051 863 The effect of indexing specificity on retrieval performance. 



ED 052 213 Attitudinal correlates of children *s speech characteristics. 
Pinal Report. 

ED 052 2k9 Effect of the response format on the differential measurement 
of traits in the Thorndlke Dimensions of TemEperamental 
Personality Invent or y- 

ED 052 299 A feasibility study in determining individual practice 

profiles of physicians as a basis for continuing education 
of these physicians utilizing a postgraduate preceptor 
technique. 

ED 052 k99 The development of cognitive performance criteria for use 
by secondary school teachers. Final Report. 

ED 052 713 Black and nonblack youth: Finances and college attendance. 

ED 052 7^9 An investigation of entrancie characteristics related to '^ypes 
of college dropouts. Final Report. 

ED 052 863. Academic performance and school integration: A multi-ethnic 
analysis. 

ED 053 063 The study of student behavior in science as a result of 
modification of certain identifiable teacher behaviors. 
Final Report. 
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ED 053 099 Assessment of the inipact of the teacher corps program 
at the Itoiversity of Miami auid participating schools in 
Southern Florida. 

ED 053 113 A no-grammar approach to sentence power: John C. Mellon •s 
sentence-combining games. 

ED 053 163 Interaction "between college effects and students* aptitudes. 

ED 053 187 The genetic components of verbal divergent thinking and 
short term memory. 

ED 053 3^4-9 Management practices of cotton producers in LauderdiEuLe 
County, Tennessee . 

ED 053 kl3 Washington, D.C. drug seminar for school personnel and students; 
Aug. 31-Sept. hf 1970. Evaluation Report. 

ED 053 5^9 The development of a measure of state epistemic curiosity. 

ED 053 6^9 Estimating parents* contribution to college costs. The 
accuracy of three measures of succeeding year family net 
income. 

ED 053 713 Class size on community college economic education. 

ED 053 8^9 The organizational climate in schools having high concentrations 
of Mexican-Americetns . 

ED 053 887 Model Programs: Reading Yuba County Reading-Learning Center, 
Marysville, California. 

ED 054 187 Research in art education: The development of perception 
in art production of kindergarten students. Final Report. 

ED 0514- 387 Survey for comparison of effectiveness of home management 
with residency versus home management without residency 
laboratory 1966-68. Final Report. 

ED 053 987 Development of a more flexible^ physical science laboratory 
program for non-science majors with superior higji school 
science backgroimds. Final Report. 

ED 03k 287 Communicative dimensions of mother-child interaction as they 
affect the self esteem of the child. \ 
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APEENDEC B 
LETTER FOR HOTIAL CONTACT WITH 
EDUCATIONAI DECISION MAKERS 



This letter was used in only about kO^ of the cases. In the 
other cases, initial contact was made by phone. The phone message 
was- similar to the content of this letter. 



\ ■ 

\ 



\ 



ERIC 



Purdue University 



SCHOOL OF HUMANITIES. SOCIAL SCIENCE AND EDUCATION 
LAFAYETTE. 'INDIANA 47907 



ED U CAT I ON 



Dear Colleague: 

We 6tre vmdertaking a study of the quality and usefulness 
of the research information disseminated through Research in 
Education, an abstracting service of ERIC (Educational 
Resources Information Center). 

W^e need educational decision makers to read and evaluate 
these research reports. We are able to pay a token honorarium 
of $15.00 for a review of a short paper (which would take 
about one hour to review) and iip to $30.00 for a longer report 
(which would take perhaps two hours to review) . We shall 
attempt to match the reports with your areas of interest. 

We shall ask you to review one of the reports and return 
yoiar analysis to us within a week to 10 days or so. If you 
are able to participate, we would appreciate it if you would 
return the enclosed formis now in the self addressed envelope. 

Thank you for your cooperation. 

Sincerely, 



William Asher 

Professor of Education and 
^ Psychological Sciences 

WA:pw 

End. ^ ^ 
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APEENDIX C 
THE DECISION MAKERS 



Stuart A. Anderson 
Professor of Administration 
Sangamon. State University 

Jane Bebb 

Director of Pupil Services 
School City of Mishawaka 

Frederick B* Bechtold 
Assistant Sutperintendent of 

Instapuction 
Marion Community Schools 
Marion, IndiaJia . 

Larry L- Beckley 
Principal 

Crete Elementary School 
Crete, Illinois 

Lawrence Beymer 

Professor of Education and 

Assistant Dean 
Indiana State University ^ 

Mark Breitenstein 

Language Arts Coordinator 

Our Lady of Lour des Elementary School 

Cincinnati, Ohio 

Donn Brolin 

Director, Iftxdergraduate Training 

in Vocational Rehabilitation 
University of Wisconsin 

James E« Brumbau^ 
Education Consultant 
Indiana State Department of Public 
Instruction ■ 

James E« Brumbaugh 
Education Consultant 
IndiaJia State Department of Public 
Instruction 

Douglas Bryant 

Professor of Education 

North Carolina State University 



Napoleon Bryant, Jr* 

Assistant Professor of Education 

Xavier University 

■\ 

Naomi S. Clark 
Teacher 

Bond Hill Elementary School 

James Clemmons 

Director of Reading 

Forest Hills School District 

Louise Cuyler 

Project Director, Head Start 

and Child Development Programs 
of the Greater Cincinnati Area 

Community Action Commission 

Harriet Darrov 

Director of Continuing Education 

for Women 
Indiana State University 

Richard Dickson 
Adult Program Coordinator 
Wabash Center 
Lafayette^ Indiana 

Frank DiSilvestro 
Educational Consultant 
Indiana State Department of 
Public Instruction 

Robert A* Eaverly 
' Principal 

Sayre. Junior High School 
Philadelphia, Pa- " 

Kenneth W- Ehle 
Consultant in Mathematics 
Indiana State Department of 
Public Jiistruction 

John L- Eick 

Assistant Professor of French 
Xavier IMiversity 



ERIC 
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Lee Elliott 

Assistaxit Professor of Psychology 
Xavier Tftiiversity 

Charles F. Elton 

Professor of Higher Education 

University of Kentucky ^ 

Edward Eustice 

Director of Graduate Studies 
Olivet Netzarene College 

Joseph L. Felix 

Associate Directoi^ for Research 

and Evalixation 
Cincinnati Public Schools 

Sister M. Floriamne 
Director, School of Nursing 
St. Elizabeth Hospital School 

of Nursing 
Lafayette, Indiana 

Lloyd E. Frohreich 

Assistant Professor of Educational 

Administration 
University of Wisconsin 

James Gaf fnejr 

Assistant Dean of Graduate School 
Xavier University 

Robert G. George 

Coordinator of Instructional systems 

Technology 
Indiana State Iftiiversity 

John Gerard 

General Manager, Seed Grain Division 
Syier, Inc. 
Plymouth,* Indiana 

Dave Gilman 

Professor of Education 

Indiana State University 

Melvin Goldberg 
Research Associate 

Institute for Educational Development 

A. C, Gondring 
• Retired Elementary Principal 
and Teacher 
Perry Central Community School 
^ Corporation 
ERsIC^^opold, Indiana 



William J. Hamrick 
Assistant Superintendent for 

Instruction 
Vigo County School Corporation 
Terre Haute, Indiana 

Beverly Kankenhoff 

Practical Arts and Continuing 

Education Director 
New C0«tl3 Area Vocational 

School and Chrysler High School 
New Castle, Indiaiia 

Robert H. Helmes 

Dean, College of Continuing 

Education and Summer Sessions 
Xavier IMversity 

Marjorie W. Hesler 

Assistant Professor of Speech 

Communication 
Indiana State Ifaiversity 

Michael D. Hinkle 
. Principal 

' Joseph F. Tuttle Jr. High 
Crawfordsville, Indiana 

Philip J. Hobbs 

Assistant Chairman, Division cf 

Education 
Indiana-Purdue Regional Center 
Indiajmpolis, Indiana 

James Hogan 

Director of Curriculum . 
Forest Hills School District 
Cincinnati, Ohio 

Donald Huff 

Assistant Director, Student Affairs. 
Indiana Vocational Technical 

Institute 
lAfayette, Indiana 

Charles A. Hvinter 

Coordinator, Sociological Research 
Dallas Independent School District 

John A. Hutchinson 
Curriculum Consultant 
Indiana State Department of 
Public Instruction 
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Jm Jacobs 

Director, Research and Dey'elopment 
Cincinnati Public Schools 

Rudolf Jacob sen 

Principal 

Cruft School 

Terre Haute, Indiana 

Frank W. Jerse 

Professor of Educational Psychology- 
Indiana State Uaiversity 

Simon 0* Jiboku 
Social StX3dies Supervisor 
^iiidiana State Depaurtment of 
Public Instruction 

Bruce C . Landis 

Coordinator for Teacher Education 
Indiana State Department of 
Public Instruction 

Conrad Lane 

Supervisor of Student Teachers 
Ball State University 

Harold W. Lang 

Director, State-Wide Testing 

Dallas Independent School District 

Jene M, Galvin 
Co-Pomder 

New Morning Commiinity School 
Cincinnati, Ohio 

Hot ley Laricenschlager 
Diirector of Laboratory School 
Indiana State Itoiversity 

Terry 0, Lowe 

Consultant for Counseling and 

Guidance Services 
Indiana State Department of i 

Public Instruction 

David H . McGaw 
Biology Program Director 
West Lafayette Higlri School 
West Lafayette, Indiana 



C« Eugene McLemore 
Director of Admissions 
Hanover College 
Hanover, Indiana 

Prank W. Medley 

Adjunct Associate Professor of 

Foreign Languages 
Indiana State l&iiversity 

Mescal Messmore 
Language Arts Consultant 
Indiana State Department of 
Public Instruction 

Leon Met 

Project Coordinator, Student 

Leadership Development Center 
Cincinnati Public Schools 

Joel Mllgram 

Assistant Professor of Education 
Ifaiversity of Cincinnati 

Rudolph P. Ml;/meman 
Director of Elementary Education 
Tippecanoe School Corporation 
Lafayette, Indiana 

Jim E, Montgomery 
School Library CoD?5Ultant 
Indiana State Department of 
Public tost ruction 

Don Moore 

Art Consultant 

Indiana State Department of 

- Public Instruction 

Marjorie Morehouse 
Remedial Reading Teacher 
Lafayette School Corporation 
Lafayette, Indiana 

Stanley R. Morgan 
Assistant Svcperintendant , 
Salt Lake City Schools 
Salt Lake City, Utah 

Stephen D, Mueller 
Guidance Counselor 
North College Hill Junior-Senior 

High School 
Cincinnati, Ohio 
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George Nagle 
Vice Principal 

Lake Osvego Elementary School 
Lake Oswego, Oregon 

Earl E. Nelson 

Director of Elementary Education 
Lafayette School Corporation 
lafayette, Indiana 

Henry W. Nelson 
Dean of Students 
Vlheaton College 
Wheaton, Illinois 

Patricia 0* Connor 
Research Associate 
School of Dentistry 
University of Michigan 

George H. Olson 

Consultant on Evaluation 

Dallas Independent Scliool District 

Jajnes B. O^Neill 

Assistant Director 

Center of Economics Education 

University of Delaware 

Willieun R. Osmon 

Dean> Student Administrative 

Services 
Indiana State Uhiversity 

Frederick K. Packer 
Director, Instructional Materials 
Center 

Coos County Intei^nediate Education 

District 
North Bend, Oregon 

Barbara Pashos 
' Reading Consultant 
Indiana State Department of 
Public Instruction 

John P, Pohlman 

Director, Lodge Learning Laboratory 
Xavier -University 

Harbison Pool 

Project Director, Project ORBIT 
Oberlin City School District 
Oberlin, Ohio 



Alton V. Potts 
Regional Director 
Tippewa Technical Institute 
Lafayette, Indiana 

Gerald L. Quatman 
Professor of Psychology 
Xavier University 

Heitry F, Raichle 
Director of Research 
Pinellas County School System 
Clearwater, Florida 

James Roland 

Director of Special Education 
Urbana Piiblic Schools 
Urbana, Illinois 

Herbert L* Ross 
Associate Professor 
School of Business 
Indiana State Iftiiversity 

John Rothwell 

Assistant to Dean - Assistant 

Professor of Economics 
Xavier University 

JoAnne S. Rowe 

Instructor in Home Economics 

Indiana State University 

Kenneth F. Scheurer 

Chairman Department of Education 

Xavier University 

Roman J.. Schweikert 

Associate Professor of Education 

Xavier University 

Dorothy June Sciaxra 
Assistant Professor of. Child 

Development 
Iftiiversity of Cincinnati 

Bernard T- Shirk 
English Supervisor 
Gory Schools 
Gary, Indiana 



Annie Sims 
Director 

Lower Price Hill Coimaunity Day 

Center 
Cincinnati, Ohio 

Dennis H. Sorge 
Federal Prograjns Evaluator 
Scottsdale Public Schools 
Phoenix, Arizona 

Walter L. Sullins 

Director, Center for Educational 

Research 
Indiana State Itoiversity 

Carl T. Swift 
Superintendent 

Pike County School Corporation 
Petershurg, Indiana 

Tom Tegarden 

Coordinator of Sicpportive Services 

for Adult Education 
Indiana State Department of 

Public Instruction 

Alice Trese 

Montessori Pre -School Instructor 
St. Augustine* 2 School 
Cincinnati, Ohio 

Christine Ann Trusler "~ 
Elementary Teacher 
Northwest Local School District 
Cincinmti, Ohio 

Mark Van Horn 

Drug ajid Narcotic Consultant 
Indiana State Department of 
Puhli'^ Instruction 

John William VanKerk 
Language Lahoratory Director 
Indiana State IMiversity 

Gerald L. Verland 

Division of Program Research 

and Design 
Cincinnati Public Schools 
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Ivan^D. Wagner 

Director of Planning Evaluation 
Indiana State Department of 
Public Instruction 

E. Dewey Wahl 
Staff Director 
Industrial Management 
Piirdue University 

William J, Wehster 

Director, Research and Evaluation 

Dallas Independent School District 

Mary Wine 

Welfare Rights Coordinator 
Santa Maria Neighborhood House 
Cincinnati, Ohio 

'Thelma M* Wootton 
Director 

Wells Memorial Library 
Lafayette, Indiana. 

Robert E. Wubholding 
Adjunct Associate Professor 
Guidance Department 
Xavier \lhiiversity 

Stanley M- Yahroff 
District Associate 
Institute t"or Educational 
Development 

Marshall A. Moore 
Director of Secondary Guidance 
Richmond Community SchooJ.s 
Richmord, Indiana 
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APEENDEC D 

THE INITIAL LETTER TO THE RESEARCH SOCIALISTS 



This letter was sent to as many as possible of the research 
specialists in Weuidt's (1967) study. 
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\ Purdue Univers[ty 

SCHOOL OF HUMANtTJES. SOCfAL SCfENCE AND EOUCATfON 
LAFAYETTE. INDIANA 47907 



DEPARTMENT OF EDUCATION ViaXOh 1, 1972 




Dear 

In 1962, paxticipated in an AERA stxuiy to examine the 
quality of published educational research literature. That 
study was under the direction of David Ryans and Edwin Wandt, 
and ms reported by AERA as, "An Evaluation of Educational 
Research in published Journals" (Mmeo, 1967), and in Wandt's 
Cross-Section of Educational Research (1965) . 

Edward VockeU and I have recently received funding for 
a U.S.O.E* Small Grant to undertake a similar study 01" the 
unpublished research disseminated through Research in Education , 
the abstracting service of ERIC (Educational Resources Information 
Center) . 



We voulii like to have the same set of reviewers as the I962 
AERA study. However, this time we can pay a token honorarium 
of $15.00 -for a review of a short report (which would take 
about one hour to review) and up to $30.00 for a longer report ' 
(which would tsUce perhaps 2 hours to review). 

We will ask you to review one of the reports and return 
your analysis of ii:s qtiality within a week to 10 days or so. 
We would appreciate it if you would return the enclosed post- 
ccurd now indicating your availability. We shall again attempt 
to match the reports with your areas of interest. 

Tliank you for your cooperation. 

^ Sincerely, 



I WAtpw 
L^9_„ Enc. 

{ERIC 



William Asher 

Professor of Education and 

Psychological Sciences 
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APEEaroix E . 

THE EDUCATIOKAL RESEASCH SPECIALISTS 



Gale R. Adkiris* 
Harold H. Abelson 
Kenneth E. Anderson 
William Asher 
Nancy Bay ley 
Carl Bereiter 
John E. Bicknell 
Walter R. Borg . 
Russel W. Burrifi 
William R. Carricker 
John B. Carron 
Werrett Wallace Charters 
David L. Clark 
Thorns D. demons 
John F. Cogswell 
Desmond L. Cook* 
Francis G. Cornell 
John T. Dai ley 
Henry S . Dyer 
Ruth E. Eckert 
Max D. Englehart 
Daniel D. Feder 
Donald W. Felker 
John C. Flanagan 
Benjamin ^IVuchter 
N. L. Gage 
George L. Gropper 
Egon G. Guba 
Elizabeth Hagen 
Ray M. Hall 
BerJ Harootimian 
Dale B. Harris 
Albert Hleronymous 
John L. Holland 



Kai Jensen 
Felix F. Kapstein 
Patricia M. Itoy 
Evan R. Keisler 
Beri: Y. Kersh 
David J, Klaus 
Dorothy M. Khoell 
Russell P. Kropp 
Philip Lambert 
Charles.R. Langmoir 
Gerald V. Lanhholm 
D. Welty Lefever 
Roger T. Lennon 
Elizabeth Stelner Maccia 
JaDoes B. Macdonald . 
Walter H. MacGintle 
Samuel T. Mayo*^ 
Ernest McDaniel 
Wilbert J. McKeachia 
John D. McLeil 
Harold E. Metzel 
Carol Miller 
William C. Morse 
George J. Moixly 
Claude L. Nemzek 
Victor H. Noll 
David B. Orr 
R. Travis Osborne 
Ellis B. Page 
Herman J. Peters ^ 
Beeman N. Phillips* 
Douglas Porter 
Thomas J. Qviirk 
William Hablnowltz 



William Reitz 
James H. Ricks 
Helen M* Robinson 
Herman Roeramich 
GcLLe W. Rose 
Philip J. Runkel* 
Einar R. Ryden 
Gilbart Sax?^ 
Dale P. Scannell 
Alice Yeomans Scatea 
Richard E. Schutz. 
C. Winfield Scott* 
Saul B. Sells 
Robert S. Soars 
Seth Spaulding 
William H. Stegeman 
George G. Stern 
Lawrence Stolurow 
Kalmer P. Stordahl* 
Doixald Thifitlewaite 
George G. Thompson 
Robert M. W. Travers 
Arthur E» Traxler 
William W. Turnbull 
Richard Turner 
R. Keith Van Wagenen 
Donald J. Veldman 
Norman E. Wallen 
Alexander G. Wesman 
Merlin C. Wittrdck 
Asahei D . Woodruff 
J. Wayne Wrightstone 



^ Read a second document 



• -. 

ERIC 



APEEUIDIX P 
COVER LETTER SENT TO DECISION MAKERS 
WITH DOCUMEINT TO BE REVIEWED 



The instructions were in this letter and on the Information 
Quality and Uses Form, Further instructions were provided only 
upon request. 

V 
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PURDUE University 



SCHOOL OF HUM a;n ities.socialscience and education 



LAFAYETTE. INDIANA 47907 



DEPARTMENT OF EDUCA-|ON 



Dear Colleague: 

Enclosed is the research report which you recently agreed to review 
as part of ovir study on educational research dissemination* Please follow 
the following steps: 

' 1. Complete the "General Information Form" (pink sheet) : 

2. Fill in the "Familiarity; with ERIC Questionnaire" (yellow 
sheet) . bo not return to this form after you have read the 
enclosed research report. 

3. Read the enclosed report to evaluate it. Use the "Information 
Quality and Uses" questionnaire (white paper). 

Complete the "Project Certification" form (white paper). 

• . 5- Return all four of the above in the enclosed stamped envelope. 
(You can keep the research report.) The university will mail 
you a check for your honorarium sometime after we receive the 
above information. 



We have attempted to match the reports with your areas ^-f interest.. 
In soKie cases we my have failed in this attempt. If you feel totally 
unable. to review the report, you have received^ return it to us and we'll 
try to make some adjustment-. 

If you have any questions^ please contact us. 

Thank you for your participation in this s^;udy. 



Sincerely, 



William Asher 

Professor of Education and 

Psychological Sciences 



WA:pw 




Enc. 



ERIC- 



- ■ i 
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APPENDIX G 
SAl^PLE OP FOLLOW-UP TEIEQIAM 

Recently you received a report to read and evaluate as peurt of our 
"informtion Quality and Educational Decision Making" study. We would 
like to receive all the completed evaluation forms by JiUy 28. If this 
is inipossible, please either return the document to us or phone ub collect 
(317-7^9-281+5) • We will reimburse you for any additional expenses you 
incur. If your records indicate th^t you liave already returned your 
evaluation fom, or if you have any other questions , please call us 
collect immediately. 



V'illiam Asher 



APPENDIX H 
COVER LETTER SENT TO RESEARCH 
SPECIALISTS WITH DOGIMENT TO BE REVIEWED 



The instructions were in this letter and on the Form for 
Evaluating Quality of Research. Further instruhtS.i^ns were 
provided only upon request. 
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Purdue Universitv 

SCHOOL OF HUMANITIES. SOCIAL SCIENCE AND ^. 

LAFAYETTE INDIANA 47907 



NT OF EDUCATION 

\ 




Enclosed is the research report which you recently agreed to 
re^/iew as part of our study on "information Quality and Educational 
Decision Making.'* 

Please read the report and evaluate it by filling in the "Form 
for Evaluating Quality of Research." (white paper) 

\. . • . 

We have also enclosed copies, of several more detailed instruments 
for assessing the quality of educational research. Th.^se are included 
for your own use. They may be helpful to you in calli^ng your attention to 
important points which should be kept in mind while you are filling out 
the other evaluation forms. Do not return any of these forms on colored 
paper . 

Return the completed evaluation form in the enclosed stamped envelope. 
Our business office also...req.uests that you fill in the "Project Certification 
form, so that we can mail you your honorarium promptly. 

We have attempted to match the reports with your areas of interest 
In some cas^ we may have failed in this attempt. If you feel totally 
unable td~ review the report you have received, return it to us and we'll 
try to make some adjustment. 

If you have any questions j please contact us. 

Thank you for your participation in our study. 

Sincerely^ 



William Asher ' 
Professor of Education and \ 
Psychological Sciences v 



WA:pw 
■ Enc. 
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APPENDIX I 



Infonwiticn Quality and Uses Form. ^ ' 

\ 



Instructions: Please fill in the 
You are encouraged to make use of 
te available to you. If you need 
on the other side of the pages or 



information as indicated, 
any additional references whAch may^ 
additional space, feel fi.ee to write 
to add additional pages . v 
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ERIC # Reviewer, ^ 



Brief Title 
Author 



1. Brief summary: (Protlem; subjects— number and description; treatments 

or classifications; procedures; analyses; results) 



2. Educational significance (Does. the report address itself to an important^ 
educational problem? Does it have a sound theoretical tasis? What is . 
ths relation to known research?) 
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Research Design (Are there-any problems in the design of the 
research or methodology employed?) 



\ 



Rate the qjaality of this research according to one of 
the following criteria i ^ 

1 , Completely incompetent--A . "horrible example." 

2t Poor --Some serious defects, 



J3i Mediocre — Not good, not bad. 



Good ~ A few minor defects, 

\ ■ ■ ■ 

3 t Excellent ~ A model of good practice. 
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5. If you were the editor of a journal to which this report 
was submitted for publication, .would yout 

A ccept ~ Accept the article and publish it without 



major revision, 

JRevise — Ask for revisions and publish it if the 
"revisions were made, 

JReject Reject the article. 



6, Estimate the approximate point where you would place your- 
self on the continuum below with regard to knowledge of the 
subject covered by the report, 

1 2 3 4 5 6 2 

know . know a 

very great 

little deal 



-7T"Give— your-estimate^of— how-signii^icant— or--^^^ 

think a change involving the subject of this report would 
be, if implemented in your school or school system. 



very very 
minor . major 

change change 



8, What recommendations would you make with regard to the 
implementation of the information contained in this 
report? 
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APmiDIX J 



Form for Evaluating Quality of Research • 



Instructions! Please fill in the information as indicated. 
You are encouraged to make use of any additional checklists 
and references which may be available to you. If you need 
additional space, feel free to write on the other side of 
the pages or to add additional pages. 



Are you currently teaching a course in research design or 
research methodology, or have you ever taught such a course? 

Yes _No 



■I 

Please list here the approximate number of research reports 
you liave either published in professional journals or pre- 
sented at conventions or profea^ional meetings s - ■ 



ERLC 
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ERIC # Reviewer ' 

Brief Title " 

Author ' • 

1. Brief summary: (Problem; subjectS"number and description; treatments 

or classifications; procedures; analyses; results) 



4 

2. Educational significance (Does the report address ?,tself to an important 
educational problem? Does it have a sound theoretical basis? What is 
the relation to known research?) 



ERIC 
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3. Research Design (Are the problems, objectives, procedures, and the ralation- 
shipa among all three clearly and logically stated? Does the statement 
of procedures include, where applicable, information on sampling techniques, 
controls, data gathered, instruments used, and statistical and other analyses 
made?) 



Rate the quality of this research according to one of the following 
criteria: 

5 , Excellent— 'A model of good practice 

Good — A few minor defects 
3 t Mediocre — Not good, not bad 
2 > Poor — Some serious defects 
1 « Conpletely Inconipete'iat A ''horrible exasrple" 

If you were the editor of a Journal to which this report was submitted 
for publication, wou]|,d you: r 

A ccept " Accept the article and publish it without revision 

R evise Aak for revisions and publish if the revisions were made 

Reject Reject the article 
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SUPPLEMENTARY EVALUATION CHECKL3STS 
SENT TO RESEARCH SEECIALISTS 
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These were not returned and are not a part of the analysis . 
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EVALUATING ETUCATIOML RESEARCH ARTICIES (Edwin Wandt) 

Rate each characteristic below on a 5-poiDt scale as follows: ^ 

5 " Excellent; a model of good practice 

k = Good; a few minor defects 

3 «= Mediocre; not good, not "bad 

2 = Poor; some serious defects 

1 = Completely incompetent; a "horrible example'' 

1 Problem is clearly stated, 

2 Hypotheses are cleetrly steted, 

3 • Problem is significant, . 

h Assumptions are clearly stated, 

5 Limitations of the study axe stated. 

6 Importemt terras are defined. 

7 Relationship of the problem to previous research is made, clear, 

8 Research design is described fully. 

9 Research design is appropriate to the solution of the problem, 

10 Research design is free of specific weaJcnesses. 

11 Method of sampling is appropriate . 

12 Population and sample are fully described, 

13 Data-gathering methods or procedures are described. 

Ik Data-gathering methods. or procedures are appropriate to the solution 

of the problem, 

15 - Data-gathering methods or procedures are utilized correctly. 

l6 Validity and reliability of the evidence gathered are established. 

17 Appropriate methods are selected to analyze the data. 

18 Methods utilized in analyzing the data are applied correctly. 

19 Results of the analysis are presented clearly, 

20 Conclusions are clearly stated. 

21 Conclusions are substantiated by the evidence presented. 

22 Generalizations are confined to the population from which the sample 
was drawn , , 

23 Report is clearly written, 

2k ■ Report is logically organized. 

25 Tone of the report displays an unbiased, impartial scientific attitude, 

26 Conelufliohs are significant. 

27 Conclusions are relevant to the problem. 
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AN INSXRIMENT FOR EVALUATING EXPERIMENTAL 
EDUCATIONAL RESEARCH REPORTS 

Marilyn N. Sxaydam 
The Pennsylvania State tlhiversity 

Directionfl: 

Evaliiate with the nine italicized questions which follow. The quality of the 
article in terms of each questioVi should he rated on a five-point scale* The 
specifications for these five points are: 

1) Excellent: all requirements for the question are oet} nothing 
essential could he added • ' 

2) Very good: most requirements are met 

3) Good: some requirements are met 
k) Fair: a few requirements are met 

^) Poor: none or too few of the requirements are met 

Certain \ey points*' should he considered in ascertaining a rating for each 
question. These are listed helow the question, followed by adjectives which 
indicate the continuum on which the ''key poiiit" should he assessed. Do HOT 
make a response to these *'key points". They are Intended to focus the attention 
of all raters on the same pertinent aspects of eadti question. 

Please make only nine responses for each article, one for each question. 

1. How practically or theoretically significant in the problem? (1-2-3-^-5) 
a. Purpose (important-non-iTrqportant) 
b« Problem origin 

1) Rationale (logical- illogical) 

2) Previous research (appropriate-inappropriate) 

2. How clearly defined is the problem? (1-2-3-^-5) 

a. Question (operational-vague) 

b. H^othesis (es) ' (relevant-irrelevant) 

(logical-illogical) 

c. Ihdependent variable(s) (operational-vague) 

(relevant^irrelevant) 

d« Dependent variable(s) (operational- vague) 

(relevant- irr elevant ) 

3. \How well does the design answer the research questions? (1-2-3-^-5) 

a. Paradi©tt ' . (appropriate-inappropriate) 

b. Hypothesis (es) (testable-untestable) 

c. Procedures (dear-unclear) 

d« Treatments (replicable-unreplicable) 

(appropriate«inappropriate) 

et iJruMAion .: (appropriate- inappropriate) 
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If. 



5. 



How adequately does the design control variables? (l-2-3-l|*5) 

a. Independent variable(s) (lancontajninated- contaminated) 

Administration of treatment (rigorous -unrigoroui) 

c. Teacher or group factors (controlled-uncontrolled) 

d. Subject or e^g^erimenter bias (controlled-uncontrolled) 

e. Halo effect (controlled-uncontrolled) 
f • Extraneous factors (controlled-uncontrolled) 

Individual factors (controlled-uncontrolled) 

How properly is the sample selected for the design and pvirpose of the 

research? (1-2-3:^-5) 



a. Population 

b. Drawing of sample 

c. Assignment of treatment 

d. Size 

e. Characteristics 



(appropriate-inappropriate) 
(random-unspecified) 
(random-unspecified) 
( appr opr iat e - inappr opr ia t e ) 
( appr opr iat e - inappropriate ) 



6. How valid and reliable are the measuring instruments or observational 



techniques? 

a. Instrument or technique 
"l) Description 

2) Validity 

3) Reliability for population 

b. Procedures of data collection 

7. How v€ilid are the techniques of analysis of data? 

aa Statistical tests 

1) Basic assumptions / 

2) Relation to design 



b:* Data 

Treatment 
Presentation 
3) Level of significance 

If) Discussion 



(l-2-3-i^-5) 

(excellent -poor) 
( appr opr i at e - inappr opr iat e ) 
. (excellent-poor) 

(careful-careless) 
(1-2-3-^^-5) 

( Sat i sf ied-unclear ) 
( appropr iat e - inappr opr iat e ) 

( appropr iat e - i nappr opr i at e ) 
(clear-unclear) 
(specif ied-unspecified) 
(appropriate-inappropriate) 
(accurate-inaccurate) 



8. How appropriate are the interpretations and generalizations from 
the data? 



a* Consistency with results 

b. Generalizations : 

c. liaplicationB 
a« Limitations 



(l-2-3-i*-5) 
(excellent -poor) 
(reasonable-exaggerated) 
(reasonable-exaggerated) 
(noted- not noted) 
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How adequately Is the research reported? 
a« Organization 
b» Style 

c. Grammar 

d. Cosgpleteness 



(i-a-s-**-?) 

( excellent -poor ) 

(clear -vague) 

(good -poor) 

( exce llent -poor ) 
( r eplicable -unrepli cable ) 
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' INSTRUMENT FOR EVALUATING SURVEY RESEARCH REPORTS 

(Richard L. Kohr, The Pennsylvania State University) 

Directions : The following instrument is to be used for evaluating survey 
research reports within the framework, of curriculum research. It is composed 
of nine major questions which are underlined. You are to rate the quality of' 
the report in terms of each of these nine questions using the following 
5-point scales 

(1) Excellent: all requirements for the question are met; nothing 
essential could be added, 

(2) ,Very good: most requirements are met. 

(3) Ciood: some requirements are met. 
(k) Fair: a few requirements are met. 

(5) Poor: none or too few of the requirements are met, 
In determining a rating for each question, certain "key points" should he 
consideredV These are listed below the question, followed by adjectives ] 
which indicate the continuum on which the key point should be assessed. 
Do NOT make a response to these "key points." They are intended to focus 
the attention of all raters on the same pertinent aspects of each question. 
In some studies certain "key points" may be irrelevant. In such cases base 
your judgment on such "key points" as are relevant. It is also possible 
that you may think of "key points" not included among those listed under 
a major question. Where relevant, such additional "key points" may be 
used in assessing that question. There may be some instances in which 
none of the "key points" seem relevant or where the report fails to supply 
sufficient information. If this occurs, evaluate the report in terms of 
what you think should have been done and/or what information should have 
been included. 

Please make only nine responses for each article, one for each questioij. 

1 How practically or theoretically significant is the problem? 
(l-2-3-ii-5) 

a. Purpor>e (important - non-important) 

b. Problem origin 

(1) Rationale (logical - illogical 

(2) Previous research (related - unrelated) 

c. Generalizability (extensive - limited) ' 

2 How clearly defined is the survey problem? ( 1-2-3-^1-5) 

a. Objectives and procedures (specified - unspecified) 

(operational r vague) 

b. Delimitations (noted - not noted) ' 
c; Variables 

(1) Control (relevant - irrelevant) 

(operational - vague) 

(2) Dependent (relevant - irrelevant) 

(operational - vague) 

3 .....^ How relevant and how well defined is the population? 

a. Precise definition of population 

(1) Geographical limits (specified - unspecified) 

(2) Time period covered (specified - unspecified) 

(3) Sociological description (specified - unsepcified) 
(h) Sampling units (specified - unsepecified) 

b. Relevance of defined population to problem (relevant - irrelevant) 
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^ How adeqiiate are the sampling procedures? (1-2-3-U-5) 
a« Adequacy of scucpling frame 

(1) 'f'iine period covered (current - outdated) 

(2) luBiveness of defined population (complete - incomplete) 

b. Metho^ of sampling (specified - unspecified) 

c. Obtained sample (appropriate - inappropriate) 

(1) Size (sufficient - insufficient) 

(2) "Rtpresehtativeness (adequate - inadequate) 

^ How adequately are sources of error controlled? (1-2-3-^-5) 

a. Sampling error (controlled - uncontrolled) 

b. Non-response (controlled - uncontrolled) 

c. Interviewer bias (controlled - uncontrolled) 

d. Response error (controlled - uncontrolled)! 

e. Response set (controlled - uncontrolled) 

f . Experimenter bias (controlled - uncontrolled) 

g. Teacher effect (controlled - uncontrolled) 

h. ' Control variables (controlled - uncontrolled) 

i. Extraneous factors (controlled - uncontrolled) 
j. Qualifications of research personnel 

(interviewers, etc.) (controlled - uncontrolled) 

How adequate are the measuring instruments? (1-2-3-^-5) 

a. Choi6e of measurement technique(fl) (appropriate - inappropriate) 

b. Instrument(s) 

(1) Development of instrument (pretested - not pretested) 

(2) Description of administration and scoring procedures 
(clear - unclear) 

(3) Wording of statements or questions (clear - unclear) 

(k) Sequence of statements or questions (logical - illogical) 

(random - fixed) 

(5) Evidence of reliability (appropriate - inappropriate) 

(6) Evidence of validity (appropriate - inappropriate) 

(satisfactory - unsatisfactory) 

c. Rules for categorizing (specified - unspecified) 

How appropriate is the. statistical analysis of the data? 

a. Procedures of data collection (specified - unspecified) 

(careful - careless) 

b. Relation of obtained data to objectives (essential - unessential) 

(sufficient insufficient) 

c. Descriptive measures 
(1) Statistics (appropriate - inappropriate) 

(2; Evaluation of . descriptive data (appropriate - inappropriate) 
(3) Establishment of relationships (appropriate - inappr opriate) 

d. Statistical tests 

^ ^X) Basic assumptions (satisfied - unsatisfied) 
yc) Relation to procedures (appropriate - inappropriate) 
(3) Significance levels (specified - unspecified) 
e« Description of results (accurate Inaccurate) 
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How reasonable are the conclusions drawn from the data? 
a* Interpretations (consistent - inconsistent) 

(reasonable - exaggerated) 
b* Generalizations (reasonable - exaggerated) 

c. Inrplleations (reasonable - exaggerated) 

d. Qualifications 

(1) Discussion of methodological problems 
and errors (comprehensive - limited) 

(2) Alternative explanations (noted - not noted) 

(3) Other limitations (noted - not noted) 

How adequately is the research reported? (l-2-3-'+-5) 

a. Organization (excellent - poor) 

b. Style t clear - vague) 

c. Grommecr and mechanics {excellent - poor) 

d. Coznpleteness (replicable - unreplicable) 

. ' / (complete - incomplete) 

e. Presentation of statistics (clear - unclear) 
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APPENDIX L 



y General Information Form 

If you are available to participate in this research project t 
please fill in the following information and answer the brief 
items in the attached questionnaire • 



Name 
Address 

Phone 

Employment or school affiliationt 
Schools 

Positions (specific) 



Type of school I 

J unior high school 
H igh school 

' College 

(other) Specifyi^^ 



Number of years as a teacher and/or administrator! v rs , 

Degrees heldi 

less than bachelor^s 
^ Bachelor's 

M aster's 

D octorate 

(other) Specify t ^_ 

Other relevant information I , (Area of interest, etc.) 
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APPENDIX M 
TABLES AND FIGURES 



This appendix contains the tables and figures referred 
to in the Results section of this report. Several notes 
are useful in interpreting these/ tables and figures t 

(1) The directionality of rating is reversed on 
Rating Scale A, This is true both for the 
ratings of the Research Specialists and those 
of the Decision Makers. A high score on Scale 
A indicates high quality. On all other scales, 
a high score indicates low quality. 

(2) The term "Group" (G) refers to a group of 
documents , 

(3) The t^rm "Trial" (T) refer's to the repeated 
measurements on the same documents by both 
a Research Specialist (trial 1) and by a 
Decision Maker (trial 2), 
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TABLE 2 

RELIABILITIES OF THE RATING SCALES* 



Research Specialists 

Rating Scale A (Overall Rating). 56 (9) 

Rating Scale B (Accept-Revise-Re ject ) 15 (10) 

Rating Scale C (Theoretical Contribution). .30 (11) 

Rating Scale D (Practical Contribution)... .51 (11) 

Rating Scale E (Methodology) 2? (11) 

I • 

Decision Makers 

Rating Scale A (Overall Rating) 36 (8) 

Rating Scale B (Accept-Revise-Re ject ) . . . . v .2^■ (8) 

Rating Scale C (Theoretical Contribution. , .51 (7) 

Rating Scale D (Practical Contribution),.. .37 (7) 

Rating Scale E (Methodology) .48 (?) 



♦Figures in parentheses after reliability coefficients 
indicate the number of documents included in the com- 
putations , 
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lABIE h 

COMPARATIVE RATINGS OF RESEARCH SPECIALISTS 
AHD DECISION MAKERS ON RATING SCAIES 



Rating SceuLe A. (Overall Rating) 



Research Decision 
Specialists MaJfcers 





No. 


Pet. 


No. 


Pet. 


5 ■ Excellent. 


5 


5.2 


15 


16.3 


h. Good 


26 


27.3 


32 


34.7 


3- Mediocre 


22 


23.1 


29 . 


31.5 


2. Poor 


32 


33.6 


15 


. 16.3 


1 . Coirrpletely incompetent 


10 


10.5 


1 


10 


* No response 


7 




10 




= 19.99 (df = 


4) p < .001 








Rating Scale B. (Accept-Revise-Reject) 










No. 


Pet . 


No. 


Pet. 


1. Accept 


8 


8.2 


30 


32.6 


2. Revise 


37 


38.1 


39 


k2.3 


3. Reject 


52 


53.6 


23 


25.0 


. * No response 


5 




10 " 




. = 23.87 (df = 


2) p < .001 








Rating Scale C. (iheoreticaJ. 


Contribution) 










No. 


Pet. 


No. 


Pet. 


1. Excellent 


2 


2.1 


7 


7.6 


2. Above average 


13 


13.6 


28 


30.7 


3. Mediocre 


27 


28.1+ 


37 


1^0.6 


K. Below average 


37 


38.9 


11 


12.0 


5 . niicorapetent 


16 


16.8 


8 


8.7 


* No response 


7 




10 




"= 26.47 (DF = 


k) p < .001 
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TABLE h, contd 



Rating Scale (Practical Contribution) 

Research Decision 
Specialists Makers 







No. 


?ct. 


No. 


Pet. 


1. 


Excellent 


3 


3.0 


11 


11.9 


2. 


Above average 


21 


21.4 


39 


i^2.3 


3. 


Mediocre 


3k 


3h.6 


23 


25.0 




Below average 


28 


28.5 


15 


16.3 


5. 


Incompetent 


12 


12.2 


h 




* 


No response 


h 




10 





)? = 22.20 (df = h) p < .001 



Rating Scale E. (Methodology) 







No. 


Pet. 


No . 


Pet. 


1. 


Excellent 


7 


7.2 


9 


9.7 


2. 


Above average 


16 


16.1^ 


ih 


15.2 


3. 


Mediocre 


2h 


2k.7 


kl 




h. 


Below average 


28 


28.8 


19 


20.6 




Incompetent 


22 


22.6 


9 


9-7 




No response 


5 " 




10 





%- = 11.81 (df = if) p < .05 



•^"No response" includes evaluations not returned and blank responses 
on the rating scale. 



no 



TABIE 5 

MEANS AM) STANDARD DEVIATIONS ON THE RATING SCAU3S 

Mean Standard Deviation 

Research Specialists 

Rating Scale A *2 .83 1 . 10 

Rating Scale B 2.k3 .66 

Rating Scale C 3-5^ -99 

Rating Scale D 3.25 1.02 

Rating Scale E 3.^+3 1.21 

Decision Makers 

Rating Scale A .98 

Rating Scale B I.92 .75 

Rating Scale C 2.83 1*03 

Rating Scale D 2.58 I.03 

Rating Scale E 3-05 I.07 



^Hhe directionality of the rating is reversed on scale A. Ki^ scores on 
this scale indicate a rating of high quality, vhereas on all other scales 
a higher score indicates lower quality. 
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TABLE 6 

COMPARISON OP TIffi RESEARCH SPECIALISTS RATINGS 
■ ON THE ACCEPT-REVISE-REJECT SCALE WITH THOSE IN 
WANDT»S (1967) STUDY 

n X S. D. 

Wandt (1967) 125 2.21 .7^ 

Present Study *96 2.^3 .66 

t = 2.29 
P < .05 



* Several Research Specialists left this item blank. 
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RATINGS OF RESEARCH SPECIALISTS COMPARED TO 



TABLE 7 

RATINGS OF RESEARCH SPECIALISTS COMPARED TO 
RATINGS OF DECISION MAKERS 

ANALYSIS FOR VARIAdLE 1 (Rating Scale A) 
"source" M E A Nl S Q U A R E D . F - R A T 1 0 P ~ 

TOTAL . . 1.232b 171 . • 

BETWEEN 1.1678 B5 

""trials * 21.<>337 "" 1 " 20.*6?. .0001 

ERROR (T) 1.0673 _ B5 

N VALID 86 

T MEAN I 2 

. .2f80<J3 .... 3.5116. 

ANALYSIS FOR VARlAbLE ? (Rating Scale B) 

SOURCE MEAN SQUARE P.F.... F-RATjO P 

TOTAL et3765 175 

' BETWEEN ."3 73<f ~" ~ 87 

TRIALS*. 12.0227 1 26.Q3'i ,0000 

ERROR (T) ,<>480 87 

■"N~V"ALl 0" '88" " " ■ 

t" mean 1 2 

2.4310 1.9091 
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TABLE 7, oont. 



F-RATIO 



ANALYSIS FOR VARIABLE 3 (Rating Scale C) 
TSOURCE^ MEAN'sVuare D.F. 
JLOXAL. Ln-lbS Lb^ 

BETWEEN 1.1482 B4 



"trials* 

_£brclr-.jj-l 

N VALID 



28,U05y 1 

6.3^1.: 3A. 

85 



32t451 



,0000 



T MEAN 



-3,6235. 



? 



a'naLYSIS for VARSAbLF '4 (Rating Scale' D) 



_S.O.U.RCE. 
TOTAL 

'between 



Jf!EAN_.S.QyARE.. 
1.2067 



—TRIALS..:! .._ .2.5.,.yi7.5 

ERROR (T) .8980 
~N VALID 89 



.D,F.,.-...l E-RA.Tj.Q. 

177 

88 ' 
. ..L. - 28, ."^.29.- 
88 



_P-. 



T MEAN 



3.3146 



■ 2 

2.5506 
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TABLE 7t cont. 



ANALYSIS.fOR VARIABLE 


5 (Rating 






SOURCE MEAN SQUARE 


D.F.. 


F-RAllO 


P 


TOTAL 1.3397 


175 






BETWEEN „ 1.4017 


' .87. 






TRIALS* 9,b5il 


1 


8.071 


.0057 


ERROR IT) 1.1833 








N VALID 88 








t.mean 1 

3.50U0 


2 

3.0341 







I 



* Note that "Trial" (T) refers to the repeated 
measurements on the same document by the Research 
Specialists (trial 1) and hy the Decision 
Makers (trial 2). 



TABLE 8 

ANALYSIS OF VARIANCE BY EDUCATIONAL LEVEL OF DECISION 
MAKERS (PH. D. VS. NON-PH. D.) 

"group" 44 SUBjecTS, "Doouments Read by Ph. D, 

Decision Makers 

S ROUP .2 .54 SUBJECTS. Documents Read by Non=Ph. D 

Decision Makers 



._SOUR.CE. 


MEAiN . SQUARE . 


O.F. . 


F-RATin 


P 


TOTAL 


1.<J350 


169 






"^ETwEETni" 


1.1811 ■ 


84" 






GROUPS 


. 1616 


.v.. 


. . .. .135 




• ERROR (G) 


1.1934 


93 






WITHIN 


I.2na2 


85 






_.J.?.IALS*_ 


20.476t> 


1 . . 


19.621 


.dOoi 


6 BY T 


2.4026 


1 


2.302 


.i?91 


"ERROR (T) 


1.U436 


83 






6 MEAN 


1 

3.1923 


3-1304 




11 . 


N VALID 


"39 " 


46 






't"mEan"~ 


1 

2.8U8 


2 ' 
3,5059 






■ G BY T 


1 


2 






" . 'T" 


2.9744 


3.4in3 






2 


2.673q 


..3.5870 







116 



ERIC 



TABLE 8, cont. 



'Af^ALYSlS FOR VARIABLE ? (Rating Scale B) 



SOURCE 



MEAN SQUARE 



TOTAL 

.!5LE.T..we.En_ 
GROUPS 



ERROR (6) 
WITHIN 



TRIALS 



G 8Y t 

-eRR(lR_Lr_L 



.5713 
I'll 12 



.5731 

...•.Sft32.. 
11.1264 



.2A20 



D.F. 



173 

B6.-.. 

1 



85 
.87 
1 
1 

B5. 



F-RATIO 



1 .939 



1639 



25.132 
.54 7 



.0000 
.5315 



_N,..VALI.0 



1 

2,2632 

.38 



2.1020 
49 



_I_tlEAM.. 



G BY T 

L. 

2 



1 

2.42b3 

1 

-2... .4 737. 
2.3B78 



2 - 

1.9195 



2 

2.. 0526 
1.8163 
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TABIB 8, cont. 



_A.N A L 15±S__Em. .. V. A R I A B L E . -. J _. 
SOURCE MEAN SQUARE 



TOTAL 

jletwelen 

GROUPS 
ERROR IG) 

_WIIJilN 

TRIALS * 
G BY f 
ERROR (T) 

Q..MEAN .._ 

N VALID 



T MEA.M 
_G. 8.Y. 

I 



1«U6B 

_ 1 ,1609. 

1 .^640 
■ 1.1572 
.... V. i72t> .. 
26. 7202 
.ti926 

.a645_.. 

1 

3.3205 

39 

1 

3.6l"^0 

1 

3.6*10 
3,6000 



O.F. F-RATio 
167 

„B3_ . 

1 1.26fi) 
82 

I 

.. . 9*^ _ ._ 

1 30.909 
1 1.033 

82. 



2 

3-1333 
4b 



2 

2.8214 



2 . . 
3.0000 
2,6667 



,2630 



.0000 
,3135 
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TABI£ 8, cont* 
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ANALYSIS FOR VARlAtiLE 4 
"SOURCE MEAN SUUARt 



TOTAL 



BETWEEN 

"groups 

_ERRQR__L6J. 
WITHIN 

"trials'*"" 

6 BY T 
ERROR (T) 



' •Boo 2 
. 1>2179 
1 tUao 
24. U 05 7 
.3.0547 
.8539 



(Rating Scale D) 

D.F. F-RATio 

-_1.75_ 

87 

1 .657 

.B6.. „. . 

BB 

1 ~' 28.11 1 

1 3.577 

86 



.S747 



.0000 
,0^5B7 



G MEAN 



N VALID 



1 

3.0000 

40 



2 

2.6646 
4» 



T mean 



G BY T 
1 

2~ 



..3.29b5. 
1 

3.22bo 
3.3542 



2 

.2»55fi« 



?. 

2,7750 
■2.3750 
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TABLE 8, cont. 

ANALYSIS FOR VARIABLE s (Rating Scale E) 



SOURCt 




MEAN SiUARE 


" D.F, 


F-RATiO 


P 


TOTAL.. . 




1.345* 


J 73 






" BEtwEEm 




i .4042 


86 










l.i562 


1.- 


... ..822.-.. 


.6296 


ERROR 


(Q) 


1.^071 


85 






WITHIM 




1.2874 


87 






TRIALS 


* 


9 •1954 


.. _.l . 


7.767 


.0'J67 


G BY T 




2.U405 


1 


1.721 


,1900 




f T 1 










G MEAN 




1 

3.1B75 


2 . 
3,3511 






M VALID 







47 






T MEAN 




3.5057 


? 

3,0460 






G BY T 




1 


2 










3.3000 


3.0750 






.^..._..._.2.... 




3.6809 


.3.0213 







I- 



♦ Note that "Trial" (T) refers to the repeated measure- 
ments on the same document by the Research Specialists 
(trial 1) and by the Decision Makers (trial 2). 
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TABLE 9 

ANALYSIS OP VARIANCE BY INSTITUTIONAL AFFILIATION 

OF DECISION MAKER 



ANALYSIS FOR VARIABLE 1 (Rating Scale A) 

MEAN SQUARE D.F. f"-RATIO 

1.2192 169. 

1.U94 

■ 1.6973 3 1.503 

_._..1.129l 

1.2882 



SOURCE 
.TOTAL 

BETWEEN 
■"GROUPS 
_ERR0R_J5) 

WITHIN 



TRIALS * 
G BY T 



ERROR (T) 



0 MEAN 



N VALID 



T MEAN 



20.4765 
__2.5302 
1.0053 



3.0000. 
7 

1 

2.8235 



84 
3 

61. 
85 
1 
3 
81 



2 

3.1765 



34 



2 

3.5176 



20.368 
2.517 



3 

3.3500 
30 



i2187 



.0001 
.0620 



4 

2.8571 
14 



^BY .L. 
1 

2 " 
.3 



3.0000 

2.9U8 
3.0333 



2.0714 



3.0000 Research Organizations 
3.4412 Colleges & Universities 
3.6667 state & Local Systems 



3.6429 Individual Schools 



121 



TABIE 9» cont. 



ANALYSIS FOR VARIABLE ? (Rating Scale B) 



SOURCE 
_TO.TAL.__ 



BETWEEN 



MEAN SQUARE 

.5790 

.5775 



GROUPS 
_ERROB._tSL 
WITHIN 



TRIALS * 

G....BY._T.__. 



ERROR (T) 



6 MEAN 



N VALID 



T MEAN 



1 

2 

_3 



5805 



ll.637«> 

.?1*7.. 

.4352 



I 

.2.21^3 



1 

2.4253. 
I. 

2.1429 
2.4063 
..2*4848. 
2.4667 



.1250 3 
-.5939 83 



D.F. F-RATio 

_ .173 - 

86 

' ."211' 



• 8890 



87 



1 
3 
83 



26.744 
2.102 



• 0000 
.1048 



2 3 
2«2I88 2.1212. 

32 33 



2 

K9080. 



4 

2«1333 
15 



2.2857 Research Organizations 
2.0313 Colleges & Universities 
-l-.7.57_6 — S t a t a & Local. Systems •- 
1.8000 Individual Schools 
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TABLE 


9» oont. 


ANiALvSTS Fan VARIABLF 


3 (Rating bcale G; 


' SOURCE 




MEAN SQUARE 


D.F. r-RATiO P 


"total"" 




l.i599 


167 


BETWEEN 




. 1.1471 


83 


SROUPS 




,8821 


3 .762 .5213 


"~ ERROR 




1.1570 


80 " ■ 


WITHIN 




1.1726 


84 


TRIALS 






1 33.047 .0000 


G BY T 




2.3649 


3 2.925 .0380 




IT; 




80 


0 MEAN 




1 

3.5.7_14 . 


2 3 4 
-.3*2500 3.1333. . 3,1000 

32 30 15 


N VALID 




7 

— ■•■ 


— 

T MEAN 





1 

3.6071 


2 

2.8095 . . 


3 BY T 




1 . 


_ c 


1 




3.4286 


3t7l43 Research Organiz;ations 


' Z~~ 




3.5938 


2,9063 Coileges &. Universitiei 


3 




3.5667 


. „2.,70pp . .State & _Lpcal Systems 



3«80U0 



2.4000 Individual Schools 
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TABLE 9, oont. 
ANALY SIS F OR VARIABLE .4 (?ating_ Scale D) 



SOURCE 


MEAN SQUARE 


D.F. 


F-RATlO 


P 


"TOTAL 


1.1774 


175 






__BETWEEN . 


1.2131 


97 






GROUPS 


.0459 


3 


.037 


.9900 


errqr~Tg)' 


1.2540 


84 






WITHIN 


1.1420 


88_ 





. ... . 


TRIALS* 


24.0057 


I 


30.312 


.0000 


" G BY "T 


3.3232 


3 


4.196 


• 0063 


ERROfi <T). 


.7920 


84__ 






G MEAN 


1 

3.0000 


2 

2.9412 


3 

2.9063 


4 

2.90OO 


N VAHD_ 


7 


34 


32 


15 


T MEAN . 


1 

3.2955 


2 

2.5568 






n RY T 


_J 


. ..2..:.. ■. 






1 


2.7143 


3.2857 


Research Organizations 


2 


3.2353 


2.6471 


Colleges and 


Universities 


3 


3.3438 


2.4i)afi_ 


State and-Xocal- Systems 


4 


3.6000 


2.2000 


Individual Schools 
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TABIE 9f cant. 



_A NALms_EOft- ftl A « LE (Rating S.cale_.E ) 



SOURCE MEAN SQUARE 

"total ' 1.3285 

..BETWEEN 1..J.702. 

GROUPS •5262 
ERROR (Q)~ 1,4007 
-WITHIN. ;.1.£87* 



TRIALS* 
G BY T 



10,1379 
2.0842 
1,1519 



D.F. 

._.fl6._ 
3 
83 

1 

3 

83..- 



f-raTio 



,376 



fi.QOl 



.7739 



.0042 
.1504 



G.._ME/IN.. 



.N-_V.ALIXL 



3.0714 

. _7 _. . 



2 

3.2794 

-34 



.. .. 

3,2097 

—31- 



4 

3.4333 

15._.. 



-T . MEAi^.. 



G BYjT 
1 
2 



3 
4 



1 

3.50S7 

2,8571 
3,4706 
3,4516 
At 0000 



2 

3,0230 



3,2857 Research Organizations 
3,0882 Colleges & Universities 
2,9677 State & Local Systems 
2.6667 ..In^iividual Schools 



* Note that "Trials" (T) refers to the repeated measure- 
ments on the same document by the Research Specialists 
(trial 1) and by the Decision Makers (trial 2), 
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TABLE 10 

ANALYSIS OF VARIANCE BY ClfiAillNGHOUSE 



"GROUP 1 


5 


SUBJECTS, 




GROUP 2 


8 


SUBJECTS. 




GROUP 3 


4 


SUBJECTS. 




~"QR0'UP~4"~" 


:w 


SUBJECTS.' 




GROUP 5 


...12 


SUBJECTS. 




GROUP 6 


5 


SUBJECTS. 




GROUP 7 ~' 


'~" b 


SUBJECTS, 




GROUP a . 


5 


SUBJECTS. 




GROUP 9 


10 


SUBJECTS. 




"group 10 


" 6 


SUBJECTS. 




GROUP 11 


9 


SUBJECTS. , 




GR0UP12 


8 


SUBJECTS. 




•_ ANA LYSIS. 


FOR 


VARIABLE 1 




SOURCE 


MEAN SQUARE 


D.F. 


TOTAL 




1 .ciseo 




GROUPS 




1 #2156 


n 


ERROR ( 


G) 


l.ii628 


55 



F-RATIO 



.963 



.5108 



G MEAN 



1 
5 

3,0000 
9 

?,8000 



2 

2.8750. 
6 

a.'tooo 

in 

2.7500 



3 
7 

2,6667 

11 
3,5000 



4 

3.2500 

8 

1.7500 

12 
3.1250 
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TABLE 10, cont. 



ANALYSIS FOR VARIAoLF ? 

.SOURCE SIEAN... SQUARE,. D.f* F-RATiO 

TOTAL ♦'►30y 77 

GROUPS .1>50B 11 1.6^1 

..ERROR.. <G). .3942 . . 66 



.i049 



6 M£AN 



1 

2.8000 

5 

P.6364 



P.BOOO 



... 2 

2.37S0 

b 

2.HOO0 

-.-W 

2.4noo 



_ 3 

2.5000 

7 

2.4000 

U 

2.0000 



. 4 
2.0U00 

. 8 

P.5000 

12 
2,0000 



_A NA L.YSI S_E0J3-....V A Rl A fi L E . . 
SOURCE MEAN SQUARt 

TOTAL t'iSbb 

_ GROUPS 1.1716 

ERROR (G) .9202 



D.F. 

76 
11. 
65 



F-RATiO 



i.<;73 



.2b94 



6 MEAN 



1 2 

2 0.0.0 3.37.50...- 



5 

.3.7273 
9 

-3.55S6- 



6 

3.H000 : 

in 



3 

-.4.0 000,. 
7 

4,0000. 

11 
.3.2222 



.._2..5UU0., 

fl' 
4,2500 

12 
3,2857 
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TABLE 10, cont. 



analysis for variable 4 
"source mean square 

TOTAL 1.0500 
GROUPS l,^2b*3 



ERROR (6) 
G MEAN 



1.053H 



D.F. 

78 
11 
67 



F-RATIO 



.<=>74 



.b£12 



1 

3.8U0'6" 
5 

3, 09 119 
9 

3.1000 



■ ?. 

3,3750 
6 

3.60 00 
2.^000 



3 

4,0000 
7 

3.16b7 

_ 11 _ 



4 

3,0000 
8 

4, oo.no' 

... .12 
3*0000 



ANALYSIS. FOR VARIABLE .5 

SOURCE viEAN SQUARE D.F. F-RATin 
"to T A L ~ 1'. 472b' \ 77' ' ' 

.._Qf^..9yPS 2.0937 n 1.5?y 

ERROR (G) 1.3690 66 



.1420 



G MEAN 



4.2U00. 
5 

3.1fll8 
9 

3.7000 



. 2 
_ .3 . 0J.0 0 

b 

'►.2000 

10 
3.6000 



3 

4,2500 



3.333^ 
U 

.3.2222 



4 

2.0000 

8 

4.2500 

12 

..3.2857. 



TABLE 11 

ANALYSIS OF VARIANCE BY SPONSORSHIP OP REPORTS 
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I 



GROUP i 11 SUQjfcCTS. AERA Convention Papers 

,.^^-.9^*tJ 9 SUBJECTS. Smaller conventions and minor 

journals ^ 
GROUP 3 4 SUHjtCTS. Educational Testing Service J 



GROUP 4 5 SUBJECTS. Masters or Doctoral Theses 
' J3R_OyP_5 73 SU8JECTS,. Other 

ANALYSIS FOR VARIABLE 1 (Rating Scale A) 
•SOURCE MEAN..SQUARE . Q.F. F-RATjO P 

TOTAL l»dZb7 94 

"groups 2.a097 4 " 2.430 .0S>26 

._EMPR L6.) 1 rises 9o 

Q._M.E_AN .. .1 2 3 4 5 

3.3UU0 1.8571 3.2500 3.4000 ?.797l 

.N_.VALlp 10 7 4 5 69 

ANALYSIS FOR VARlAbLE~Y ^Rating Scale' B) 

SOURCE MEAN SUUarE D.F. F-RATm P 

TOTAL .-+356 96 

"groups" ■ ".3303"'" 4 ,.750 .5626 

'■ ERROR (G) .4401 92 



G MEAN_ 1 i? , 3 , , 4 S 

2*2Td7 . 2.6667 2.7500 2. 2500 ?.42i)3 

N VALID 11 . 9 4 4 69 
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TABLE 11 » cont. 

ANA LYSTS Ffi R VARIAbL F (Rating Scale C) 

SOURCE MEAN SQUARE D.F. F-RATiO P 

TOTAL .VPSl" 94 

_.G8.Q.U£5 56 1. 3 , 5 1 1 ,0105 

ERROR (S) .B990 9n 

Q MEAN i 2 3 ^ ^ 
3..;jb:36. /L.500.0 2.500O 3.2a')0 3.5522 

N VALID n e ^ 5' 



ANALY.SI.S..EO.R..VARI ABUE. iRatin^L Scale Dl..._ 

SOURCE MEAN ScjUarE D.F. F-RATio P 

TOTAL ~ V.useo 97 

_J5LR.0.UPS ..ridbb .4 .735 .b727 

ERROR (0) 1.U696 93 

G MEAN 1 2 3 4,5 

.__ S.OOgQ ... .3^5556 -2.7500 3.6000 3.26i)9- 

N VALID 11 9 4 . 5 ft4 

ANALy.SJ.S_JEOB.. V^AHI.AtiLE -5 (Kat^Jlg. A9ale . E ) 

SOURCE MEAN SQUARE D.F. F-RATm P 

TOTAL 1 .4588 96 

_GROUPS. 3.1512..; . .4... 2.275 .0662 

ERROR (SJ 1.3852 92 

Q MEAN .12 3 4 5 

. 3.,,3..636. 4'.. 0000. 2.2500 .2.6000 . , . 3.5294 

N VALID XI — — ; 9—, 4 5 6*^ 



1 



1 

TABLE 12 

ANALYSIS OP VARIANCE BY DECISION SITUATION 



ANALYSIS FOR VARIABLE 1 (Rating Scale A) 

^OURCe~ MEAN'SQUArE D, F. ~ ' F-RAT 10^ ' P " 

L_TOTAL 1»1861 _ Ul .. _ 

BETWEEN 1.0177 70 
GRb'lJPS ^.3 8 0 3" 3 ~ • 36 V~ ~ .782 4 ' 
ERRO_R__LO)_ 1.0462 67 _ 

WITHIN 1.3521 71 

Trials*' 19.0423 1 19.789 ,0001 

___Q.BY_J ^.i6l9 3„: 4.325 .0078 

ERROR (T) .9623 67 

G MEAN 1 2 3 4 

_ 3.30^3 .3.1765 3.0909 . 3.1000 

N VALID 23 17 11 20 

T MEAN 1 ? 
2.8169 3.5493 . . 

_Q _B Y .T 1 2 Infonnation Change 

1 (n=25) 2.5217 4.0870 High Major 

2 (n=19) 3.1H6 ■ 3.2353 High • Minor 

3 (n=ll) 2.8182 3.3636 Low Major 
^ (n=2i^) 2.9o"oo 3.3000 Low Minor 
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TABLE 12, cont. 



1 ANALYSIS FOR 


VARIABLE 


2 (Rating Scale B) 




SOURCE MEAN SQUARE 


U • r • 




P 


__.TOTA| 


.S98B 








BETWEEN 


.6099 


f J 






""'""oRdu'p's" 


' .b9'22 


3 


f970 




_ ERROR f (3 1 


.6106 


._ .. ..70_ 







WITHIN 


.5)878 


74 






TRIALS* 


10.2770 


1 


25.781 


f ) 0 n n 
t u u u u 


1. S. BY L. 


1.7729 


. . 3 


4.447. 


.0067 


ERROR (T). 


.3986 


70 




— . 


G MEAN 

-- - - 


1 

2.0400. 


-2.1176 


- 2.0909 


4 

2.3095 


N VALID 


25 


17 


11 


21 


T MEAN 


1 

2. 4064 . 


2 

1,8784. 






9 BY T 


. 1.. 


2 


_In.lormatiQn 


Change 


1 (n=25) 


2.5200 


1.5600 


High 


Major 




2.11/6 


2.1176 


High 


Minor 


3 (n=ll) 


_2.4545.._ 


_-.1.7273_ 


Low 


.Major 



4 (.n'=24) 2»4762 2,1429 Low Minor 
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TABLE 


12, cont. 






-_ANALYSlS-.JroR 


VARIABLE . 


3^ -(Rating^csile Q) ... 




SOURCE MEAN SQUARE 


D.F. 


F-RATIO 


P 


TOTAL 


1 t 104U 


141 






...BETWEEN 


1.03BO. 


70 






5R0UPS 


1*2453 


3 


I .210 




ERROR (G) 


1«0287 


67 






WITHIN 


1*U90 








TRIALS * 


^23,6901' 


i' 


_ 33,555 


• 0000 


S..BY.J. 


4,007.3 


. 3. 


5.678 . 


• 0019 


ERROR (T) 


• 7058 


67 







G MEAN 




1 

.3.0000 . 


2 

3.4000.. 


3 

-- 3.2500 


4 

3.3095 


N VALID 


25 


15 


10 


21 


T MEAN 


I 

. 3.6197 . 


2 

.. .2.8028 






3 BY T 


1 


2 


„I.nf omation . 


Change 


1 (n=25) 


3.8000 


2.2000 


High 


Major 


2TH^19) 


~ 3,5333 


3.2667 


High 


Minor 


3(n=ll) 


3«500p. 


.._ 3.00DO. 


..Low 


Major 


4 (n«2i^) 


3.5238 


3.0952 


Low 


Minor 



ERIC 
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TABLE lZ$ cont< 



_A.N.AUYSjLS_!LOR VARIABLE l (Rating Scale _D ) 



SOURCE 

Total 

JETWEEN 

GROUPS 
""ERRbsTTGy 
W ITHIN 

TRIALS * 

ERROR (T) 



MEAN SQUARE 
'"'1.1724 

1.7822 
1.1176 

l..«2000_ 

22.<>267 
4.8638 
.7462 



D.F. 

149 
.74 
3 
71 
75 



F-RATIO 



1 

3" 
71 



1*595 



30.054 
6.518 



1970 



.0000 
.0009 



Q MEAN 



N VALID 



2.6600 
25 



2 

2.9167 

.18 



3 

3.1364 
11 



4 

3.0714 

21 . 



T MEAN 



1 

3.2933 

.1 



2 

2.5200 



J3 BY,1 .1. __ . 2. . It, formation 

1 (n=25) 3.40U0 1,9200 High 

^.....^.^..^.^^^-^^ "2,88b9 i" 2,9444 High 

3 (n=ll) 3^.12.13 ! 54S5 Lim 



4(n=24) 3.2857 



2,8571 



Low 



-Change 
Major 
Minor 

...JWa j or - 
Minor 



TABLE 12 1 cont. 



-_ANALY3XS-E0R-V.ABIABLE— s (Rating Scale E) 

SOURCE MEAN SQUARE O.F, F-RATiO P 

""total 1.2670 U7~ 

__flE.tWEE-N !• 20 88 -73 

GROUPS 1,2054 3 •997 ,5993 

r~ ERROR (G) 1.2090 70 

WITH IN 1,3243 ^7A 

TRIALS* 8.7568 1 8.234 .0056 

g¥y~T '~ 4,9339 ~ 3 4.64O .0054 

ERRQfi._LUL l.,0634. JTO—l 

_Q_MEAN. .. 1 .. 2 3 . 4 

3.0400 3.3824 3.3182 3.3810 

N VA LID 25 . _ ..17 . 21 .. 

LjL_NlEAf4..„.. 1 2 

3.5000 3.0135 

I f 

i ^Q~BY~T 1 2 Information Change 

ljn=25.) 3.7200 2.3600 High Major 

; 2 (n=19) 3.3529 3.4118 High Minor 
i 3 (n=riT ^3.4545 3.1818" Low~ Major 
4 (n=24) 3.3810 3.3810 Low Minor 



*Note that "Trial" (T) refers to the repeated measure- 
ments on the same document by the Research Specialists 
(trial 1) and by the Decision Makers (trial 2), 
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TABLE 13 

SUIflMARY OF THE CHARACTERISTICS OF THE ERIC DOCRVIENTo 



1970* 1971 
(Vockell & Asher, 1970) (Present Study) 



Total Documents 

Examined 7^5 6l7 

Number of Docu- 
ments with 155 203 
Research 

Percent of Docu- 
ments with 20,8 32,9 
Research 

Percent of Re- 
search Docu- 39.3 32,5 
ments Rated 
Good or Ex- 
cellent 

Percent of Total 
Documents Sup- 
ported by Good 6,7 5,0 
or Excellent 
Research 



* The 1970 sample was based on the abstracts from the first 
ten months, of that year. Since this was a pilot study, 
the criteria for classification were not as carefully 
supervised as in the present study. Each document was 
classified only once by one of sixteen reviewers. No 
reliability check was attempted during the pilot study, 

** Based on the research reports actually available. Reports 
not available to the reviewers were excluded from this 
computation. 

Based on the total number of documents originally 
examined. Thus this figure indicates the percentage 
of the original documents which are supported by 
empirical data. The other reports either contained 
no data or were unavailable. 
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5.0 

3.5 
3.0 

2.5 
2.0 

1.5 
1.0 



Documents 
evaluated by 
Ph. D. 
Decision Maker 



Documents 
evaluated by 
non-Pli!,/ 3, 
Decision i^laker 



= Research Specialists 
=» Decision Makers 



Exact means for each group can be found in Table 8. 

Also note that the directionality of Rating Scale A 
is reversed. A high rating on this scale indicates 
high q.uality in the docuuent. On all other scales a 
higher rating indicates lower quality, 

FIGURE 1 

INTERACTION OP LEVEL OP EDUCATION OF DECISION MAKER 
WITH DISCREPANCIES IN RATINGS ON RATING SCALE A * 
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3.0 

2.5 
2.0 

1.5 
1.0 



Documents Documents 

evaluated "by evaluated "by 

Ph. D. non-Ph. D. 

Decision Maker Decision Maker 



= Research Specialists 
a Decision Makers 



Exact means for each group can "be found in Table 8. 

FIGURE 2 

INTERACTION OF LEVEL OP EDUCATION OF DECISION MAKER 
WITH DISCREPANCIES IN RATINGS ON RATING SCALE B ♦ 
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5.0 

^.5 
i+.O 

3.5 

3.0 

2.5 
2.0 

1.5 

1,0 



Documents 
evaluated by 
Ph. D, 
Decision Maker 



. Documents 
evaluated by 
non-Ph. D. 
Decision Maker 



= Research Specialistti 
B Decision Makers 



* Exact means for each group can be found in Table 8. 

FIGURE 3 

INTERACTION OF LEVEL OF EDUCATION OP DECISION MAKER 
WITH DISCREPANCIES IN RATINGS ON RATING SCALE C ♦ 



5.0 

3.5 
3.0 

2.5 
2.0 

1.5 

1.0 



Documents 
evaluated by 
Ph. D. 
Decision Maker 



Documents 
evaluated "by 
non-Ph. D. 
Decision Maker 



Research Specialists 
Decision Makers 



* Exact means for each group can "be found in Table 8. 

FIGURE^ 

INTERACTION OP LEVEL OP EDUCATION OP DECISION MAKER 
WITH DISCREPANCIES IN RATINGS ON RATING SCALE D ♦ 
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5.0 

-^.0 

3.5 
3.0 

2.5 
2.0 

1.5 
1.0 



Documents 
evaluated by 
Ph. D. 
Decision Maker 



Documents 
evaluated by 
non-Ph. D. 
Decision Maker 



B Research Specialists 
« Decision Makers 



* Exact means for each group can be found in Table 8, 

FIGURE 5 

INTERACTION OP LEVEL OP EDUCATION OP DECISION MAKER 
WITH DISCREPANCIES IN RATINGS ON RATING SCALE E * 



Ik2 



5.0 
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FIGURE 6 
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FIGURE 8 
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FIGURE 9 
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Exact means for each group can be found in Table 12, 
~; FIGURE 11 

INTERACTION OF DECISION SETTING WITH DISCREPANCIES 
IN RATINGS ON RATING SCALE A ♦ 



ll»8 



3.0 

2.5 
2,0 

1.5 




l.C 



High' 
Information 
Major 
Change 



High 
Information 
Minor 
Change 



Low Low 

Information Information 
Ma j or Minor 
Change Change 



= Research Specialists 
B Decision Makers 



Exact means for each group can be found in Table 12, 

FIGURE 12 
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Exact means for each group can be found in Table 12 , 

FIGURE 13 
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Exact means for each group can be found in Table 12. 

FIGURE 14 

INTERACTION OF DECISION SETTING WITH DISCREPANCIES 
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Exact means for each group can te found in Table 12 . 

FIGURE 15 
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